* Change Ingress version from v1beta1 to v1 for K8s 1.22+ compatibility
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Minor fix
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
PR adds a new executortype which supports running containers as functions. New CLI under functions is added to create container as functions.
Co-authored-by: Harsh Thakur <harshthakur9030@gmail.com>
Co-authored-by: Sanket Sudake <sanketsudake@gmail.com>
1. Support for adding labels and annotations to functions & environment via fission CLI through create & update command.
2. Change ensures labels and annotations assigned to the environment would reflect on pods created via executor type pool manager and new deploy.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Retrieve pod metrics only if metrics server is running
Currently we query pod metrics every 30 sec which floods executor logs,
added check which confirms if metrics server is running then only we start
querying pod metrics for identifying CPU utilization.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Fixed couple of typos and misspells with Go CI
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Remove unnecessary conversions with Go CI
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* add multi env and multi k8s support on integraation tests
* test workflow
* test workflow
* test workflow
* remove test branch name
* correct kind action with image and version
* use same kind version as per kind images
* use OS as ubuntu-latest
* fetch upstream
* add .github/workflows to paths
* resolve conflicts
* make kindimage appear first on workflow
Co-authored-by: Sanket Sudake <sanketsudake@gmail.com>
* Correct release name in release script for non helm install
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Increase test timeout to 20 minutes
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Enable codeql for master only
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Added GCP PubSub Image, enabled Keda by default and updated versions of all Keda connector images
Signed-off-by: Vishal Biyani <vrbiyani@gmail.com>
* Fixed the hyphens, it's been long since I have written Helm templates :)
Signed-off-by: Vishal Biyani <vrbiyani@gmail.com>
* Fixed the hyphens, it's been long since I have written Helm templates :)
Signed-off-by: Vishal Biyani <vrbiyani@gmail.com>
* Update charts/fission-all/values.yaml
Co-authored-by: Harsh Thakur <harshthakur9030@gmail.com>
* Update charts/fission-core/values.yaml
Co-authored-by: Harsh Thakur <harshthakur9030@gmail.com>
Co-authored-by: Sanket Sudake <sanketsudake@gmail.com>
Co-authored-by: Harsh Thakur <harshthakur9030@gmail.com>
* Add a SharedIndexInformer for services and deployments to NewDeploy executor.
This brings the NewDeploy executor behaviour into sync with GenericPoolManager behaviour
by caching Kubernetes services and deployments used in per-request function validation.
* Create informers in executer by executor label
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
Co-authored-by: James Sinclair <irgeek@btc.com.au>
* Enable prefix based routing
Signed-off-by: Harsh Thakur <harshthakur9030@gmail.com>
* Optimize checking condition
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Disable few tests
* disable router modification for now
* run code generator
* Collect fission dump in CI
* Enable all tests back
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Remove unwanted code
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Add prefix support at more places and couple of todo's
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Few more changes
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Improve function trim support
* Support for prefix based urls in fission function test
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* multi route for fission function test
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Improve validations in trigger creations
* Adjust leading / in url from fission
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Change suburl to subpath for function test
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
Co-authored-by: Sanket Sudake <sanketsudake@gmail.com>
* Add a warning if spec is applied on a dirty worktree
* Update go dependencies after rebase
* Use the console package for showing the warning
* Fix merge conflicts
* Handle errors and use better messages
Co-authored-by: Sanket Sudake <sanketsudake@gmail.com>
Since we recently migrated all fission examples to fission/examples repository, we need to change
respective references.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
We are facing issue in CI for multiple PRs due to mlock. It depends on kernel version or ulimit. Trying to switch to latest go version from 1.14 and see if it helps!
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
Series are sorted internal by "order by", but the set of all series returned is not. Tried some ways to sort completely on database, found no way .. client side sort works at least
Co-authored-by: Vishal <vishal-biyani@users.noreply.github.com>
Adding once only execution pattern to pool manager based functions. This is for use cases where you don't want to share the samne pod for another execution instance!
In test_mqtrigger.sh and test_mqtrigger_error.sh tests, we timeout program
if not received right output. Ensuring we capture timeout signal and do program
cleanup accordingly. Also printing due to which signal program failed.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
Moved out environments and examples out of main Fission repo to their own repo. This is to allow changes to environment releases to be independent of main Fission releases.
Websocket event support for cleaning up pods only after WS connection is terminated. The support for websocket is right now in the environment itself and router simply acts as a proxy for WS communication!
Increase the default concurrency to 500 from 5 set in the function. Usually, it is good to error on the higher side and if needed, users can always fine-tune it to their needs.
This feature enables routing more than one request to a pod at the same time. This is the first draft of the work and might involve more optimizations later.
Passing information of function from router to executor is more efficient than calling the K8S API. This change does that instead of passing only metadata and then executor calling the K8S API again.
A ready pod which can be specialized was fetched for every function earlier, this has been changed to a queue and cache implementation in client-go to improve performance.
Poolmanager when tested at high load had some issues and this PR fixes one set of them which were found so far.
Co-authored-by: Vishal <vishal-biyani@users.noreply.github.com>
Concurrency in the pool manager allows specializing pods concurrently based on a specified limit.
Co-authored-by: Vishal <vishal-biyani@users.noreply.github.com>
Attribute enableServiceLinks can be now configured in PodSpec so that environment variables are not injected in function pods and DNS is used for discovery
This PR let the controller returns the detail error message
of kubernetes error which makes users understand what's
happening instead of meanless error msg.
This PR changes the behavior of controller API which
wrongly exposes sensitive data to the client. Now, the
API only returns success if secret/configmap exists;
otherwise, an error will be returned.
In some cases, creating a package and querying the package info
immediately the Kubernetes API server will return "not found"
error. So retry the query again after some time.
In some cases (#1384), users may want to access the file in the directory of deploy archive, however, we use the function UID name as the directory name and make it difficult for users to use as they don't know the real path of the directory.
This PR uses the fix name "deployarchive" to make things easier.
If the user applies package YAML file has no status field,
the package won't be able to be compiled or deployed due
to lack of status. This PR aims to add a check at buildermgr
to set initial package status to those packages.
Executor wrongly deletes role bindings if the user
creates an environment in the reserved namespaces. This
PR is a quick fix to solve the problem by checking if
an environment is under reserved namespaces.
This PR adds a fake controller client for local CLI operations like
offline spec generation or for unit test purposes. The fake client
now only implements the "Version()" function and more functions
will be implemented once we figure out how to achieve the goals
mentioned above.
The root cause of the problem is that eagerPoolCreator
tries to create the deployment when the poolmanager
is trying to delete it. To avoid this, start eager pool
creator after executor starts serving requests.
Adding fission spec list functionality. It lists Functions, Environment, Packages, HTTPTrigger, MessageQueueTriggers, Canary Config, TimeTrigger, KubeWatch created by an application specification.
The 3rd party package version used by the go server
may be different from the one in the user's source code
and will cause plugin version mismatched. Hence, we
should never import any external packages except the
Fission or built-in packages.
This PR replaces all 3rd-party packages with built-in packages.
This PR adds an interface for controller API client, it allows us to
implement mock API client for unit testing with ease and we are
able to generate spec file without accessing the real Fission server.
The pod template is embedded inside the deployment. So if
the pod annotation contains instance-id, the deployment
will get updated and thus triggers a rolling update whenever
a new executor starts which is unwanted.
After this PR, poolmanager will patches instance-id when a
pod is chosen for a function.
For newdeploy, unlike poolmanager manages the lifecycle
of function pod directly, newdeploy is only responsible
to create the deployment so we append instance-id to top-
level controller (deployment) only.
We used to update timestamp in the deployment environment field
in order to trigger a rolling update when the function referenced
resources get updated. However, use timestamp means we are not
able to avoid triggering a rolling update when executor tries to adopt
orphaned deployment due to timestamp changed which is unwanted.
In order to let executor adopt deployment without triggering a rolling
update, we need an identical way to get a value that can reflect res-
ources changed without affecting by time.
To achieve this goal, the sum of the resource version of all referenced
resources is a good fit for our scenario since the sum of the resource
version is always the same as long as no resources changed.
When a new executor starts up, it adopts the orphan kubernetes resources created
by the old executor instance. However, the adopted resource won't reflect the changes
come with the new executor, for example, the fetcher image inside won't be changed.
To solve this, executor updates the resource spec (HPA/Deployment/Service) with the
latest resources spec. By doing this, we can prevent the inconsistency between resources
created by different executor instance, also minimizes the impact on users.
Previously, once the executor is deleted for reasons (like upgrade or cluster scale-in),
the new executor deletes all existing resources created by the old executor and creates
new one. This mechanism becomes a problem when there are requests connecting to the
existing pods. Also in the worst case, the cluster may not have enough resources to create
new pods and cause service downtime.
This PR let each executor type adopts existing resources before starting the executor
API services, and so the alive connections won't experience failure. However, the requests
send to the function that doesn't have alive function pods will still fail due to the
executor is in bootstrapping.
The pool manager keeps terminating function pod periodically even there are
traffic to the function. The root cause is that executor, poolmgr, newdeploy
manage their own functionServiceCache separately. And when router taps a
function, executor updates the access time of the function service entry in its
own cache without notifying executor types to do the update as well. Hence,
the access time of function service entry in poolmanager cache never gets updated.
Due to the access time never gets updated, the idle pod reaper in poolmanager
then thinks the function pod is in idle state and recycle it.
This PR removes the cache in executor itself, and when router tries to tap a function,
executor will call executor type to tap the function and update access time.
The router taps function service one by one which is inefficient and
increases the burden of executor. This PR aggregates all requests into
one to solve the problem mentioned above.
When a function is created before the creation of the environment it's used, the newdeploy will not be able to create kube objs. Hence no function service record is inserted into the cache.
When getFuncSvc is called, the newdeploy tries to find the record in service cache in order to create kube objs with the same name used in previous kubeobjs creation. However, due to no record in the cache, a NotFound error is returned directly and causes the problem. To solve this, we use fn meta UID to ensure we always get the same obj name instead of getting it from the cache.
If the context of request is closed before ReverseProxy finishing writing
a huge response body to the response writer, the client will only receive
a truncated response body.
To solve this, move the context cancel after ReverseProxy finished.
To keep archive creation implementation simple and prevent any
confusion, we decided to remove `--keepurl` flag and embed URL
directly without downloading the file from it . In this way, we can
ensure consistent behavior in either package creation or spec file
creation. Also, it increases the portability of spec file.
1. The records are stored in redis which is not migratable to another cluster for the testing purposes.
2. Some of the requests fields are not recorded.
3. People should consider using https://github.com/buger/goreplay which is an existing mature and well-tested solution for testing purposes.
The function implementations of controller client package are
inconsistent. This PR lets functions reuse the functions
that already implemented and able to set additional headers to
request.
When `fn test` failed to retrieve logs from the function
pod, it turns to query logs from the log database. However,
the logdb type flag is not set to `fn test`, hence the logdb
type is empty (even without default value) and cause `fn logs`
returns an error due to unable to find the corresponding log
database type.
This PR adds logdb type flag to `fn test` to resolve the problem.
Jaeger preserves the trace data of router healthz endpoints
which is not so helpful for monitoring function metrics and
consumes storage to store such trace data.
This PR uses `GetStartOptions` to examine the request's path and
check if it's a request to healthz endpoint. If yes, skip and not
to trace it.
To follow 12-factor app rules, make jaeger collector endpoint
as an environment variable instead of CLI args. It's easier to
replace the value in different deployments. Also, we can utilize
valueFrom to get value from the configmap.
The helm pre-upgrade check failed due to the program tries to
check whether a multierr is a nil pointer instead of errors it contains.
This PR changes the way the program checks multierr in order not to block the upgrade.
Helm treats the single line "extraCoreComponentPodConfig" in
values.yaml as a key value pair config.
```
extraCoreComponentPodConfig:
```
And when a user tries to install helm with providing config
like following, it failed.
```
extraCoreComponentPodConfig:
nodeSelector:
serverless: fission
```
The root cause is that helm treats config in default values.yaml
and providing config as two different types of config, which is
key-value vs. value table. And so it's unable for helm to merge these
two configs even with the same key name.
This PR comments out the single line config in default values.yaml
to prevent the issue.
Before this PR, CLI functions fatal out when encountering error
instead of returning it. Such behavior makes it hard to reuse
the functions nor writing unit tests. This PR aims to make functions
return errors instead of error out.
Istio sidecar proxy blocks all requests sent through the reverse proxy
to the target service if the request.Host is not properly set to the
internal target service host. This PR sets the target service hosts
before establishing the proxy for the client in order to pass the
Istio sidecar proxy check.
Tensorflow Serving supports three kinds of API: predict, classify, regress. In order to fully support possible API types, this PR adds code to split the entry point with separator ":" to get model name and the API type separately. And append the API type to proxy URL when receiving the requests.
$ fission fn create --name t1 --pkg <pkg name> \
--env tensorflow --entrypoint "half_plus_two:regress"
The utility function GetMetadata uses wrong flag text to get the resource namespace and could cause the wrong results returned from the API server. This PR changes the function signature that allows users to pass in the flag text in order to get the correct value.
In the case of large files, it takes a long time for the user to download
the source from the URL and upload it to StorgeSvc through CLI.
This PR allows a user to use URL as the function source when creating a function
and provides a new flag "--keeparchiveurl" to let the user to decided
whether the CLI should download the file first or store the file URL in the
archive directly. If "--keeparchiveurl" is true, then no checksum will be
generated, it's the user's responsibility to ensure the file won't be changed.
When a function with executor type newdeploy got created, Newdeploy
is expected to create deployment/HPA/service for it and insert serviceEntry
to the cache for later use. Once clients call the function, newdeploy returns
the serviceEntry to the router.
However, the log shows that the newdeploy was unable to find the entry and
prints "Resource not found - key 'xxx' not found". The root cause is that the
informer controller instead of processing items in parallel, it dispatches XXFunc
to process items one by one. So if there is any problem during the creation of the
kubernetes resource, it takes a longer time to process the next item and hence
the serviceEntry was not inserted before clients call the function.
This PR lets the newdeploy to process items in extra goroutines instead of blocking
the process loop. It's a workaround to solve the problem above, we should consider
using workqueue to solve it in the future.
Encoded path means the URL path contains encoded string like "/foo%2Fbar".
Gorilla/Mux by default doesn't enable encoded path support, you need to enable
it when initializing the router.
This PR adds a new environment variable USE_ENCODED_PATH to the router
deployment to enable encoded path support.
If a user deploys fission in the namespace which is different from the one in the single YAML file generated by helm, fission components won't be able to talk to each other due to the wrong namespace appends after the service address.
This PR adds --namespace when generating the YAML file to prevent the mismatch problem.
ReplicaSet generated by Deployment distributes pods to nodes based
on nodes resource usage, which means in some cases the newly created
pods may be assigned to only a few nodes. The requests will go to the
same node if there multiple pods on it and may increase the node
resource usage and overall request latency.
To solve the problem mentioned above, this PR allows users to deploy
the router as DaemonSet so that we can distribute requests across all
nodes for better workload distribution and lower latency.
The merge function executor used wasn't merge container correctly, and it
didn't merge all fields in spec except volumeMount & Env which confused people.
To apply the user-configured container correctly, this PR changes the way of merge
and follows rules:
1. Slices are merged and return an error if the elements in the slice have name conflicts.
2. Maps are merged, the value of map of dst container are overridden if the key is the same.
3. The rest of the fields of dst container are overridden directly.
In 1.9, kubernetes prompted Deployment to apps/v1 and finally deprecated
extensions/v1beta1 support in 1.16. Users will see the following error message
in log when fission components try to submit Deployment to the k8s API server
and specify extensions/v1beta1 as API group.
error: unable to recognize "deployment": no matches for kind "Deployment" in version "extensions/v1beta1"
This PR updates call to k8s API server and apiKind in chart files to avoid fission failure in 1.16.
Router readiness probe failed due to kubelet cannot find router healthz endpoint.
This is caused that the healthz endpoint is added only when there is any update request
send to the updateRouterRequestChannel, and makes router failed.
This PR sends an update request right before the router is started to avoid the problem.
The router prints error no matter what error type it is.
It's useful for troubleshooting, however, it also prints
the context canceled error, which means that users abort
request before reply and its really normal nowadays. Also,
the router returns 502 if error is not nil and may confused
client if it's a timeout error.
To solve these problems, this PR adds an error handler to
reverse proxy to examine the return error and change the
status code when needed.
Previously, the spec doesn't update the package status if nothing
in the spec file changed. Due to this, the failed package will
always stay in the failed state.
This PR adds a check to see whether a package is in the failed state.
If yes, then changes the state to pending for builder manager to rebuild it.
Previously, a user has to delete HTTP triggers point to
the same function one by one. This PR adds new flag --function
to list & delete commands. So that user can delete or list all
triggers with the same function. However, this feature is not
working for triggers with canary deployment setup.
For each ingress controller, the format of ingress host,
path and annotations are different. To support different
kinds of controller, this PR adds new ingress config field
to http trigger spec. A user can set annotations, host and
path based on the type of underlying ingress controller
with CLI.
Command example:
fission route create --name foo \
--url /foo/{bar} --function foofn --createingress \
--ingressannotation "nginx.ingress.kubernetes.io/ssl-redirect=false" \
--ingressannotation "nginx.ingress.kubernetes.io/use-regex=true" \
--ingressrule "*=/foo/*"
* Add specializationtimeout flag to function create/update
* Set specialization timeout of 120 seconds if not present
* Add default newdeploy timeout for rest of the test cases
* Comment out specialization timeout in validations for compatibility
* Add warning if specializationtimeout is lower than default value
This fix allows the empty repository tag in values.yaml file. The changes are done in the templates of deployment and fluentbit where repository tag will be added in the docker image name only when specified with some value in case of empty repository tag image name contain the name of image and tag only.
* Config and secret change to invoke function update
* Added recycle function for pool manager as well - where it recycles the specialized pods
* Switched to rolling update of pods and env variable based change instead of deleting pods in new deploy executor
This PR removes not so useful logs and changes most of Info level
log to Debug/Error level in hot path while preserving some of them
that is helpful for troubleshooting.
Currently, only one CI build can be started at the same time due to we only have only one testing cluster. This PR aims to enable multiple concurrent builds can be triggered to reduce the waiting time for CI builds. The major changes listed below:
1. Each build generate unique images for testing and debugging
2. Check whether the testing cluster is being used by another build.
3. Cache docker image & go mod after build finished
4. Improve dockerfiles for reusing docker build cache
* Use terraform to manage ci cluster. Upgrade ci cluster to 1.13
* Upgrade helm version
* Do not break the test if no previous resources
* Do not skip install helm if helm is old version
* Add support for setting nodeSelector/affinity/tolerations on all deployments of fission components.
* Add extraCoreComponmentPodConfig helm value to allow setting of affinty/tolerations/nodeSelectors/etc on core pods
The prometheus server may not come up in a short time or when a cluster doesn’t have a storage class config. Because of this, the connectivity test will increase controller initialization time and so the controller will fail at readiness probe detection. Since we don’t want to delay the time that controller comes up also canary config manager prints error log when it failed at connecting to prometheus server, remove the connectivity test here.
Consolidates the addition of the fetcher container to the pod into a new type FetcherConfig and takes care of serviceAccountName if not set. Also added PreStop lifecycle handler if pod set TerminationGracePeriodSeconds
Currently the implementation of the CLI (or any Fission component) specifies any timeout on
the fission fn test functionality. If a user function or environment does not specify any
timeout the command will stall. Nor is there any parameter to set a timeout if the user
wanted to.
This change sets the sensible default for the timeout to 30 seconds, to avoid new
users having to figure this out the hard way. The alternative could be to set the timeout to
0 (no timeout).
Future work: extend this timeout parameter to a global flag, and support
timeouts on all CLI commands.
From go release history (https://golang.org/doc/devel/release.html#go1.11)
and travis ci build history, looks like 1.11.7 and 1.11.8 contains unknown
bugs that breaks build. To avoid this, fix go version to 1.11.6 for now.
* Added support for --plugin parameter in Fission CLI
This fixes the issue that renaming fission to fission-XYZ would cause a fork bomb, due to the CLI recursively calling itself to discover plugins.
As of K8s 1.8, RBAC mode is stable and backed by the rbac.authorization.k8s.io/v1
API. Using v1beta1 API version for RBAC based objects fails helm deployment
of fission on minikube
It should be possible to keep zero pool size when the environment is created so that there are no idle resources for a newdeploy function. Fixes: https://github.com/fission/fission/issues/1104
This removes fluentd in favor of using fluentbit, which is lighter (in
memory usage) and seems to be more actively maintained.
Fluentbit's config file format is different from fluentd's. It also
doesn't support the same record modification stuff that fluentd
supports, so we have to change the influxdb query slightly. This
means that after an upgrade, the new CLI may won't work for querying older
logs. Hopefully, this slight breakage is acceptable; if users
really need older logs they can use the older CLI.
The root cause of the issue was introduced by PR https://github.com/fission/fission/pull/1009/files .
To be short, even the CRD of environment was delete, it still takes time for executor (poolmgr) to destroy env pool. In our cases, the previous test creates an env and delete it when test finished, then the next one creates the same name env, but failed to create pool due to the deploy name conflict. So the executor selects the pod from the first created env pool. Then, executor starts to delete the env pool, and makes the pod state became Termination state. To fix this problem, a unique name of deployment will be returned after this PR to prevent the name conflict.
* Send the error message to user when enabling canary feature fails.
* Fix controller doesn’t reply canary error when promClient tries connect to invalid prometheus server
* Changed Kafka topic name validation so that capital characters are also allowed
* Removing support for \ in Kafka topic name
* Using raw string some specific sequences
Added support for Kafka headers to be passed and retrieved from functions. The headers and supported and work only for Kafka version 0.11.0.0 and higher.
* Refactor specialization process
* Separate readiness and liveness probe to different routes
For newdeploy, readiness probe should check whether a fetcher specializes env container successfully or not. In this commit, fetcher returns the actual state of current specialization status instead of returning 200ok directly.
Archives with multiple files should have all files in the inputs.
Also clean up some inconsistent variable naming -- archive name,
archive file path, and archive input files were all stored in
incorrectly-named variables.
* Remove version from release name since it contains illegal chars for names
* Put version in release name for uniqueness, but remove invalid chars
* Remove prometheus=false yaml
* Optimize router response time by adding update lock
In 0.9.2, the router sends multiple GetServiceForFunction requests to executor
to get the service URL. However, the response time of executor will increase
under high-concurrency situation due to too many requests are waiting for
processing.
To solve the problem, an update lock was added to the router. All of goroutines belongs
to the same function need to grab the update lock before sending the request.
Only the first goroutine which gets the update lock is allowed to send request.
In this way, we reduce the burden of executor and lower the failure rate.
A simple test for Kafka integration which runs locally provided cluster has Kafka and Fission install has Kafka integratiom enabled (mqtrigger-kafka deployment present).
Improved some of warnings/messages shown to the user when using `fission spec` command. The `spec validate` command now shows appropriate warnings instead
This PR introduces extensibility to the Fission CLI. The principle behind the design of this extensibility is taken from the approach git uses: the main binary (fission) calls other binaries (fission-workflows) - which it discovers using simple prefix-rules - based on specific commands (fission workflow <command>).
* Fix CLI failed to set up port-forwarding when multiple controller pods exist in the same namespace
* Check pod healthy state before establishing the connection
Java environment builder with support for Maven builds. Default build comman runs `mvn clean package` and picks up target/*with-dependencies.jar for function.
This changes image pull policy from Always to IfNotPresent. The previous
policy forbid customer to use the exact image existing on K8S Node.
And it should be OK to use fixed policy here. Customer need not set policy by
their own.
The Round Tripper parameters of timeout, keep alive time and Max retry
times is configured in router system environment.
And they are set to Round Tripper when router service initializing.
This setup new nested structure `tsRoundTripperParams` to transfer them.
This line warns the user that dependent environment doesn’t exist, and presents the command line
for creating an environment.
When the variable `envNamespace` is specified, the `envns` option should be available as well.
* Python Environment: add gevent based WSGI server framework
This supply gevent based WSGI server framework to Python Environment, to
allow customer to run their functions under multi-threads.
* classify the flask based application
This set up the flask based application as individual class, to make
sure the consistent and integrity.
* Add instruction of WSGI server selection applied in Python Environment
* Python Environment starting logger: provide framework name
For certain environments, the archive should not be extracted and kept as it is (For ex. for Java the jar file should not be extracted). This change enables an environment level flag to control this behaviour.
For archives bigger than 256K, the Storage service was called from the client side, this needed few environment variables to be set. This change uses port forwarding to achieve the same and does not need environment variables to be set.
This use constant to instead hard code for ArchiveLiteralSizeLimit. That ensure if the limitation
value is changed in cli, the controller server can follow it. Otherwise, the condition will return
error, and make error HTTP response.
Later, we should delete it in fission-cli, and make it configurable it chart, to work in fission-
servers only.
It is mandatory (from this release onwards) for function to refer to secrets, config-maps and packages in its own namespace to ensure isolation of users sharing the same cluster.
This change runs a pre-upgrade job to verify function references for all functions created prior to this release and fails the upgrade by printing a list of functions that violate this restriction.
Ingress integration to allow the optional creation of ingress for a given route. The ingress controller needs to be set up by the user separately so that ingress path is accessible outside the cluster.
This feature allows creation of fission objects in different namespaces, in addition to retaining the existing behavior of creating fission objects in default namespace if user doesnt provide one.
It also removes cluster admin roles for fission-fetcher and fission-builder Service Accounts and grants them only those privileges that they need.
* Fixes the issue with fn test and adds relevant test cases, fixes#650
* Checking log size for invalid function
* Testing the invalid function logs a few time beforr quitting
* Disabling the test for now, linked to issue #653
The transport in RoundTripper is the default transport and there's just one object shared between different http requests and releasing the idle Connections ended up releasing the idle connections that were established with the executor service address too.
So every time a http request was made to get a service for function, the transport layer ended up having to dial a new connection (that was timing out due to various reasons).
By removing the CloseIdleConnections, we minimize the need for transport to dial a new connection to executor for every request, thereby minimizing the occurrence of dial timeouts.
The router's cache entry for a function might become stale if the pod that had the function specialized gets deleted somehow. In such a case, we'd retry getting a new service for the function from executor and retry forwarding the user request to the newly created service.
Adds a validator, invoked through `fission spec validate` and before `fission spec apply`. The validator checks for duplicate names, dangling references, and unused resources.
Also attempts to fix a race between CLI and builder. Avoids replacing a package while it's building, instead waiting for the build status to change from pending before updating it.
The cli spec dir flag name should be `specdir`, not `specs`, that should match to:
```go
specDirFlag := cli.StringFlag{Name: "specdir", Usage: "Directory to store specs, defaults to ./specs"}
```
in fission/main.go
Currently it cause the specific specs dir not to work.
```
$ fission spec apply --specdir <my spec dir>
Everything up to date.
```
Tests for updates to a function of new deployment executor type. Tests check for changes in the environment, scale, secrets etc. Also checks conversion of function executor type from new deployment to pool manager and vice versa
The Deployment definition for executor, controller, router contains readines/liveness probes. The port definition for those probes should be numbers, instead of string, i.e. port: "8888" should be port 8888.
Environment Specs so far had only an image URL to specify a container image.
This was fine for public images but fell short in a few of cases:
(a) Using private image registries
(b) Specifying environment variables (this is needed for workflows helm install)
(c) Setting a SecurityContext for the container
This change adds the Container object to both build and runtime Environments.
Compatibility is preserved -- the existing ImageURL field is still used. See the comments in types.go for the overriding rules in the case that both Container and ImageURL are specified.
Newdeploy manager now checks the existence of function service cache by function UID before trying to create a new deployment. And return the cached fsvc directly if the cache exists.
Remove the hard coded default "fission" namespace that was added in the last PR, replacing it with some logic to figure out where fission is installed and connecting to it if there's exactly one installation. If there is more than one fission installation, error out with a useful message.
Also improve the error message for a missing kubeconfig file.
This makes set up easier for new users. Users can still set
FISSION_NAMESPACE but it will default to "fission". Users can also
still set KUBECONFIG, but it will default to $HOME/.kube/config.
Also, update the post-install chart notes.txt and the install guide to
use "fission function test" as the first step after install. This
means that if there's anything wrong with the setup, the user will see
useful errors instead of "internal server error". Also, they can test
their setup without worrying about nodeports or ingresses or whatever.
All existing functionality of FISSION_URL and FISSION_ROUTER continues
to work.
This the very first step for fission to integrate with Istio, which is an open platform to connect, manage, and secure microservices. With Istio, users are able to monitor functions usage and trace requests latency through dashboards. For more information, please visit http://fission.io/docs/
setting serviceType to ClusterIP as default for fission controller and corresponding changes in fission cli to be able to port-forward the controller pod.
The fetcher needs a relatively smaller set of resources and does not have to be same as the function container/defaults. This change adds defaults for fetcher containers in function pods.
* prevent leaking packages; add new clis to list and delete orphan packages.
* Fixes after testing.
* Address review comment.
* Removing deleting orphan packages from fnUpdate and fnDelete.
* goFmt and removing unwanted code.
These commits implement support for consuming messages from an Azure storage queue to trigger Fission functions.
* Add stubbed Azure message queue implementation and modify Helm charts.
This commit stubs an implementation for an Azure storage message queue trigger
that will be completed by future commits.
It also modifies the Helm chart to add support for deploying Fission with an
mqtrigger configured for Azure storage queue triggers.
* Add Azure Go SDK to glide.
This commit adds the Azure Go SDK to glide for the upcoming work to support
Azure storage queue triggers.
* Implement Azure message queue trigger.
This commit implements a message queue trigger based on Azure storage queues.
Required message queue trigger manager environment variables:
* AZURE_STORAGE_ACCOUNT_NAME - the Azure storage account to use.
* AZURE_STORAGE_ACCOUNT_KEY - the Azure storage account key.
When creating a message queue trigger, the topic will be the Azure storage
queue to receive messages from.
* Add CA certificates to fission-bundle.
This commit adds the root CA certificates to the fission-bundle image. This
allows Fission to contact third-party APIs that use HTTPS with root CA
signed certificates.
* Add Makefile to build and test.
This commit adds a simple Makefile for building the client and bundle, running
tests, creating the Docker image, and pushing the Docker image.
This change enables users to have declarative specifications for Fission resources. Users can specify their "app" in a set of spec files, and use a new fission CLI command to "apply" these specs to a running cluster.
The new "spec" CLI also includes archiving of local source files, and a file watcher that re-builds archives and uploads them on file changes, and a package build watcher that waits for package builds on the CLI.
A "--spec" option is also added to "function create", and will be added to other resources in future changes. This option causes a YAML to be outputted to the specs directory instead of the resource being created on the cluster.
The CLI "fission spec --help" outputs usage information.
This commit solves part of the issue #52 , functions are able to access secrets/configmaps specified by the user. For now, CLI only accept one secret/configmap. For advanced users, it will be able to use YAML to declare multiple secrets/configmaps in later changes.
A newdeploy backend which uses new deployment to serve requests. This is the second phase of #193 and builds on top of changes in #384 .
* Executor layer added on top of pool manager
* Removed the external server for executor
* Minor changes to keep existing semantics as much possible
* Separating the executor vs. poolmgr backend functionality and associated data members
* Executor logic separated from Poolmgr backend completely, placeholder for new backend
* Changed references to poolmgr in tests
* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package
* Rebased from master and changed references to tpr -> crd
* Executor layer added on top of pool manager
* Executor logic separated from Poolmgr backend completely, placeholder for new backend
* Changed podName to a generic objectReference in fscache (#391)
Changed podName to a generic objectReference in function service cache implementation.
* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package
* Rebased from master and changed references to tpr -> crd
* Merged from master with latest changes
* Executor layer added on top of pool manager
* Removed the external server for executor
* Minor changes to keep existing semantics as much possible
* Separating the executor vs. poolmgr backend functionality and associated data members
* Executor logic separated from Poolmgr backend completely, placeholder for new backend
* Changed references to poolmgr in tests
* update compiling.md to use helm
* Compile instructions: changed pullPolicy to IfNotPresent (#378)
Containers will get stuck in ErrImagePull/ImagePullBackOff state otherwise
* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package
* Fetcher called when pod is created for newDeploy backend but also supports older way, this is WIP and still needs pod specialization and creating & exposing a service so the URL can be hit by end user
* WIP Specializing the POD as part of startup along with fetching
* Working specialization of a new deployment. Needs some work on caching, cleanup etc.
* Switched to service based address instead of POD address
* Minor formating issue fixed
* Added logging to pods and a readiness check, the readiness check is flaky though ATM
* Fixed some rebase issues that were failing build
* Better names for K8S objects and methods
* Switched usage of FuncSvc in backends from pod to api.ObjectReference
* Adding retry to fetcher request, for now just using default retry client which might need tweaking in future
* Switching to plain old retry, some issue in getting retryablehttp with glide import
* Removed stale executor service & deployment from previous merge
* Addressed review comments, still testing some areas
* Added types in FunctionSpec
* Resolved conflicts due to merge from executor_abstraction branch
* Added backend type on EnvironmentSpec along with operations for create/list/update, the pools are created/destroyed based on change in backend type
* Backend from types and a minor err return issue fixed
* Draft version of CPU and memory parameters added to environment
* Added resourceReq to newDeploy, though it has some issues
* Issue with resourceName fixed, now newdeploy pods also pick up resources from the environment config
* Adding scale params, removing validation on CPU params for now
* Fixed a formatting issue
* Checking if slight more delay helps in the test which is currently failing for internal routes
* The resourceList newly added in Env can not be compared by compiler, hence must use breakdown comparison instead
* Added strategy selection on client side
* Added caching, informers, delete operations for newdeploy backend functions
* Deleted a stale directory
* A simple HPA based on scale parameters, testing still WIP
* Fixed a small issue in delete function, added HPA delete too when deleting a function
* Previous merge missed the pkg flag for update fn command somehow, fixed that
* Fixed comments from review
* Changed poolmgr cleanup to be generic cleanup and moved to executor, added instanceID labels to newdeploy so that cleanup works
* Moved instanceIdLabel to types to avoid cyclic dependency
* More review fixes
* Tweaking sleep to see results
* If user does not provide poolsize, then it should not default to zero
* Switched to naming convention for now, fixed default poolsize if not provided
* Changed error return behaviour in delete fn, also changed cleanup to look based on obj type though support for additional type will need more work
* Changed check location so avoid false logging
* Test for newdeploy backend
* Adding tests for poolmgr backend
* Fixed an issue with glide dependency version, already fixed in master
* Added instanceId for NewDeploy, Initial cleanup now cleans older objects of newdeploy backend, removed eagercreate flag and instead using minScale to drive eager creation
* Moved cleanup to executor layer with cleanup for newDeploy backend, changes to use the new Cache impl
* Cleaning up pod & rs along with deployment for newdeploy backend
* Enhanced fn and env listing to show min/maxscale and resuorces respectively
* Added conditional heapster deployment and fixed a small issue with resources for fetcher container in function pod
* Addressed review comments from previous change
* Addressed some more review comments - majorly create only on NotFoundError
* Added TargetCPU as an input for scaling
* Bumped target CPU to be greater than 0 and added a default value
* Min replicas should be 1 even if the minScale is 0 when creating deployment
* Changed name from 'backend' to executorType, added additional test for minscale 0 case, changed TargetCPU to TargetCPUPercent
All functions have a pkg reference. This can be a package with either source and a deploy archives, or, a deploy archive. Everytime a function is updated, a new package is created. With archive pruner, the archives that are pointed to by old pkg reference can be deleted from the storage.
* High level spec for package pruning.
* Skeleton for archive pruning
* Adding meat 1 to skeleton.
* Adding meat #2. Separated storage service into a httpHandler component and
Storage Layer component.
* Adding meat #3. getOrphanedArchives in pruner and getItems on
stowClient.
* Restructured archivePruner methods.
* Commiting the day's work. Ready for testing #1.
* Fixing compile errors.
* Test ready. added a few logs for debugging.
* Adding a filter for getItems in stowClient.
* After testing.
* Added a test for archivePruner.
* Adding helm value pruneInterval for testing.
* Modified test.
* Final test.
* Fixing interval from seconds to minutes.
* Small change.
* Changing debugs to info.
* Removing the WIP design
* Ran gofmt on all these files.
* Fixing prune_interval as string in ENV var.
* Addressing all comments, but one.
* changing getFile method in stowClient to stream it into a response.
* All comments incorporated.
* Introducing a new flag for running archivePruner.
1. This flag is disabled for archivePruner to run in unit test.
2. This flag is enabled for archivePruner to run in production.
3. Also disabling test_archive_pruner.sh in this PR. Follow up with
next PR to enable it.
* Addressing review comments.
* Changing the command to generate a file dynamically.
* Enabling arching_pruner_test
* giving execute permissions to test_archive_pruner.sh
* Making changes of positional parameters after recent commit.
Change test case permission and removing kubectlPortForward.
* Adding debug to see why test_utils.sh passed junk pruneInterval.
* shell needs special handling for positional parameters from 10.
By default, flask turns redirects into absolute URLs. This is
undesired behaviour on Kubernetes, because the Pod's own IP address is
not really useful to any client that's not in the cluster (and even
within the cluster most clients will want to use the address of a
Service, not a Pod).
* Add readiness probe
* Remove builder manager http api interface since we don’t use/need it
* Check environment builder status and block build requests until builder is ready
* Replace deprecated api extension interface
* Add healthy check to python env
Modifies the fsCache to support a list of arbitrary kubernetes objects instead of just one pod. This allows better support for the newdeploy backend.
Also removes the byKubeObject cache index; cleanup is changed to use the byFunction index, and byKubeObject is no longer needed.
A few improvements to the integration test:
* Search for test files recursively in the test dir
* Output a test report at the end for a quick summary of what tests passed/failed/were skipped
* Add support skipping test files; skip the logging test since it's broken (#446 is tracking this)
* Switch to latest helm version
* Cleanup namespace
tested fission on openshift 3.7.0 and installed fine, without having to use openshift variables
those variables were used prior to switching to custom resource definitions, and as such are obsolete
Add package command support: This PR adds package command to CLI. package provides some useful subcommands to use such as packages CRUD and display package detail information. Also it is able to reuse existing package at function creation.
* Check function existence before creating package
* Check package existence
* Add support for downloading archive from given url
The functionality was supported before e238776bf7. Add this functionality back for more flexible usage.
* Add output flag to save archive content in specific file
* Add logs when package create/update
* Fix fnCreate requires environment argument when a pkg is specified
* Fix fnUpdate failed to update function pkg info when a package is specified
* Add package command test
* Fix wrong python test image in test script
* Remove package description fields
* Fix package command not update package but creating a new one
* Set package as pending status only when there is no deploy archive
* Rename function from fetchArchiveFromArbitraryURL to downloadToTempFile
* Use io.Copy to prevent loading all body into memory
* Revise some messages
* Allow user to update package build command
* Allow user to update package content when using function update
* Retrieve pkgName from function packageref if it’s not specified
* Fix function failed to update due to resource conflict
* Fix test case failure due to single quote
* kick ci
* Fix failed to update function packageRef
* kick ci
* kick ci
Before this change, poolmgr tried to hit fetcher just once. However if the pod has just started, fetcher may not be up yet, causing the initial request to fail with a connection refused error. This change adds a similar retry loop that we currently use on the specialize endpoint of the runtime environment container.
Create a container image with the buidler, so we can avoid copying the builder binary into the env container during the env container build.
Gather both stdout and stderr of a build into the logs.
* Add Dockerfile for builder + update release scripts
* Improve logging of buildermgr and builder
* Add support for environment-scoped buildcmd
* Fix fork/exec wd when package is a file
* Update tests to also include builder image
* Add logging to builder's fork/exec
This change adds a layer of abstraction over poolmgr. Poolmgr is now just one of the ways to turn a function into a service; other implementations will be added. The executor abstraction is a uniform API over all these implementations.
* Executor layer added on top of pool manager
* Removed the external server for executor
* Minor changes to keep existing semantics as much possible
* Separating the executor vs. poolmgr backend functionality and associated data members
* Executor logic separated from Poolmgr backend completely, placeholder for new backend
* Changed references to poolmgr in tests
* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package
* Rebased from master and changed references to tpr -> crd
* Executor layer added on top of pool manager
* Executor logic separated from Poolmgr backend completely, placeholder for new backend
* Changed podName to a generic objectReference in fscache (#391)
Changed podName to a generic objectReference in function service cache implementation.
* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package
* Rebased from master and changed references to tpr -> crd
* Merged from master with latest changes
* Removed stale executor service & deployment from previous merge
* Addressed review comments, still testing some areas
Remove the `logger` container from the logging daemonset.
Remove the outgoing call from the poolmgr to the logger. Use Fluentd's Kubernetes filter to add function name and UID to influx metadata.
This means fluentd now figures out when to start collecting function logs on its own, without being informed by poolmgr. This is great for other execution strategies, and for autoscaling, where fission isn't in direct control of function pod creation.
Also adds an integration test to make sure logging keeps working.
This fixes a bug where functionReferenceResolver returned out-of-date function metadata and caused the router to proxy requests to old function pods.
It also uses the go context package to shutdown the controller when the router is shutting down.
This contains a few different builder manager fixes:
* Update the zip file structure to avoid an extra subdirectory
* Annotate packages with what functions are using them
* Only trigger builds when there is no deployment archive
* Python environment loadpath bugfixes
* Other bugfixes
Switch Fission's storage over to the new CustomResourceDefinitions, from the deprecated ThirdPartyResources. This allows us to be compatible with Kubernets 1.8 and onwards.
This also adds a CLI tool for dumping state from an old fission version and restoring state into new CRDs.
The storage service is unaffected by this change.
closes#319
* Make default node-env use alpine, and support separate debian env. List envs in documentation.
* Copy ONBUILD base image commands into node Dockerfiles
* Copy more commands from onbuild image
* Tidy dockerfiles
Also, if you're familiar with a language that we don't support today,
adding an environment for a new language is a good task to take on.
Take a look at the Python environment as an example.
# Choose something to work on
If you're interested in working on something that doesn't have an
existing issue, feel free to file a new one.
* The easiest way to start is to look at existing [issues](https://github.com/fission/fission/issues) and see if there's something there that you'd like to work on. You can filter issues with label "[Good first issue](https://github.com/fission/fission/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)" which are relatively self sufficient issues and great for first time contributors.
- If you are going to pick up an issue, it would be good to add a comment stating the intention.
- If the contribution is a big change/new feature, please raise an issue and discuss the needs, design in the issue in detail.
There's plenty of stuff to do. If you need help choosing something to
work on, we'll be happy to suggest something depending on your
interests.
* For contributing a new Fission environment, please check the [environments repo](https://github.com/fission/environments)
### Talk to people
* For contributing a new Keda Connector, please check the [Keda Connectors repo](https://github.com/fission/keda-connectors)
At any point, from thinking about contributing to merging your
changes, please feel free to reach out to us! We're more than happy
to talk.
If you choose a big chunk of work that involves design changes to
Fission, please talk to us while planning the change; that way we can
all be on the same page before you invest too much work into it.
### Get Help.
* Drop by the [slack channel](http://slack.fission.io)
Do reach out on Slack or Twitter and we are happy to help.
* Drop by the [slack channel](http://slack.fission.io).
* Say hi on [twitter](https://twitter.com/fissionio).
* Email soamvasani@platform9.com
### Contribution Howto
* Fork this github repo
* Make a branch in your fork
* Commit your changes to that branch
* Create a pull request
# Contributing - building & deploying
## Prerequisite
- You'll need the `go` compiler and tools installed. Currently version 1.12.x of Go is needed.
- You'll also need [docker](https://docs.docker.com/install) for building images locally.
- You will need a Kubernetes cluster and you can use one of options from below.
`cmd` package is entry point for all runtime components and also has Dockerfile for each component. The actual logic here will be pretty light and most of logic of each component is in `pkg` (Discussed later)
| Component | Runtime Component |Used in|
| :------------- |:------------- |:-|
| fetcher | Docker Image |Environments|
| fission-bundle | Docker Image |Binary for all components|
**fetcher** : is a very lightweight component and all of related logic is in fetcher package itself. Fetcher helps in fetching and uploading code and in specializing environments.
**fission-bundle** : is a component which is a single binary for all components. Based on arguments you pass to fission-bundle - it becomes that component. For ex.
So most server side components running on server side are fission-bundle binary wrapped in container and used with different arguments. Various arguments and environment variables are passed from manifests/helm chart
**fission-cli** : is the cli used by end user to interact Fission
**preupgradechecks** : is again a small independent component to do pre-install upgrade tasks.
### pkg
Pkg is where most of core components and logic reside. The structure is fairly self-explanatory for example all of executor related functionality will be in executor package and so on.
```
.
├── pkg
│ ├── apis
│ ├── builder
│ ├── buildermgr
│ ├── cache
│ ├── canaryconfigmgr
│ ├── controller
│ ├── crd
│ ├── error
│ ├── executor
│ ├── fetcher
│ ├── fission-cli
│ ├── generator
│ ├── info
│ ├── kubewatcher
│ ├── logger
│ ├── mqtrigger
│ ├── plugin
│ ├── publisher
│ ├── router
│ ├── storagesvc
│ ├── throttler
│ ├── timer
│ └── utils
```
### Charts
Fission currently has two charts - and we recommend using fission-all for development.
```
.
├── charts
│ ├── README.md
│ ├── fission-all
│ └── fission-core
```
### Environments
Each of runtime environments is in fission/environments repo and fairly independent. If you are enhancing or creating a new environment - most likely you will end up making changes in that repo.
This repository contains a theme for [Hugo](https://gohugo.io/), based on great [Matcornic Learn theme](https://github.com/matcornic/hugo-theme-learn/).
Visit the [theme documentation](http://docdock.netlify.com/) to see what is going on. It is actually built with this theme.
# Main features
- Search
- **Unlimited menu levels**
- RevealJS presentation from markdown (embededed or fullscreen page)
- Attachments files
- List child pages
- Include segment of content from one page in another (Excerpt)
- Automatic next/prev buttons to navigate through menu entries
[Hugo-theme-docdock {{%icon fa-github%}}](https://github.com/vjeantet/hugo-theme-docdock) is a theme for Hugo, a fast and modern static website engine written in Go. Hugo is often used for blogs, **this theme is fully designed for documentation.**
This theme is a partial porting of the [Learn theme of matcornic {{%icon fa-github%}}](https://github.com/matcornic/hugo-theme-learn).
{{%panel%}}docDock works with a "page tree structure" to organize content : All contents are pages, which belong to other pages. [read more about this]({{%relref "content-organisation/_index.md"%}}) {{%/panel%}}
## Main features
* [Search]({{%relref "search/_index.md" %}})
* **Unlimited menu levels**
* [Generate RevealJS presentation]({{%relref "page-slide.md"%}}) from markdown (embededed or fullscreen page)
* Automatic next/prev buttons to navigate through menu entries
Feel free to update this content, just click the **Edit this page** link displayed on top right of each page, and pullrequest it
{{%alert%}}Your modification will be deployed automatically when merged.{{%/alert%}}
## Documentation website
This current documentation has been statically generated with Hugo with a simple command : `hugo -t docdock` -- source code is [available here at GitHub {{%icon fa-github%}}](https://github.com/vjeantet/hugo-theme-docDock)
With **Hugo**, pages are the core of your site. Organize your site like any other Hugo project. **Magic occurs with the nested sections implemention done in v0.22 of hugo (congrats @bep)**.
With docdock, **Each content page composes the menu**, they shape the structure of your website.
To link pages to each other, place them in a folders hierarchy
```
content
├── level-one
│ ├── level-two
│ │ ├── level-three
│ │ │ ├── level-four
│ │ │ │ ├── _index.md
│ │ │ │ ├── page-4-a.md
│ │ │ │ ├── page-4-b.md
│ │ │ │ └── page-4-c.md
│ │ │ ├── _index.md
│ │ │ ├── page-3-a.md
│ │ │ ├── page-3-b.md
│ │ │ └── page-3-c.md
│ │ ├── _index.md
│ │ ├── page-2-a.md
│ │ ├── page-2-b.md
│ │ └── page-2-c.md
│ ├── _index.md
│ ├── page-1-a.md
│ ├── page-1-b.md
│ └── page-1-c.md
├── _index.md
└── page-top.md
```
{{%alert info %}} **_index.md** is required in each folder, it's your "folder home page"{{%/alert%}}
### Add header to a menu entry
in the page frontmatter, add a `head` param to insert any HTML code before the menu entry:
example to display a "Hello"
+++
title = "Github repo"
head ="<label>Hello</label> "
+++
### Add icon to a menu entry
in the page frontmatter, add a `pre` param to insert any HTML code before the menu label:
You can define additional menu entries in the navigation menu without any link to content.
Edit the website configuration `config.toml` and add a `[[menu.shortcuts]]` entry for each link your want to add.
Example from the current website, **note the `pre` param** which allows you to insert HTML code and used here to separate content's menu from this "static" menu
Create a `_header.md` page in content folder. Its content is what you get in the logo placeholder (top left of the screen).
{{%alert info%}}**Tip :** you can add a image, a combobox with links to other documentation....{{%/alert%}}
{{%alert info%}}**Tip 2 :** look at [ extra static menu]({{%relref "extramenu.md"%}}) if you want to add links to other website in this sidebar{{%/alert%}}
Hugo-theme-docdock defines two types of pages. _Default_ and _Slide_.
* **Default** is the common page like the current one you are reading.
* **Slide** is a page that use the full screen to display its markdown content as a [reveals.js presentation](http://lab.hakim.se/reveal-js/).
* **HomePage** is a special content that will be displayed as home page content.
To tell Hugo-theme-docdock to consider a page as a slide, just add a `type="slide"`in then frontmatter of your file. [{{%icon circle-arrow-right%}}read more on page as slide]({{%relref "page-slide.md"%}})
Hugo-theme-docdock provides archetypes to help you create this kind of pages.
## Front Matter
Each Hugo page has to define a Front Matter in yaml, toml or json.
Hugo-theme-docdock uses the following parameters on top of the existing ones :
+++
# Type of content, set "slide" to display it fullscreen with reveal.js
type="page"
# Creator's Display name
creatordisplayname = "Valere JEANTET"
# Creator's Email
creatoremail = "valere.jeantet@gmail.com"
# LastModifier's Display name
lastmodifierdisplayname = "Valere JEANTET"
# LastModifier's Email
lastmodifieremail = "valere.jeantet@gmail.com"
+++
## Ordering
Hugo provides a flexible way to handle order for your pages.
The simplest way is to use `weight` parameter in the front matter of your page.
[{{%icon circle-arrow-right%}}Read more on content organization]({{%relref "content-organisation/_index.md"%}})
A basic md content page can be rendered as a reveal.js presentation full screen.
{{%alert info%}}You can, also, **embed presentation in a page** as a small box, using the [revealjs]({{% relref "shortcodes/revealjs.md"%}}) shortcode in your md file.{{%/alert%}}
## Formating
Use your common Markdown syntax you use in Hugo, don't forget, you can put html tags too.
{{%notice info %}} Special syntax (in html comment) is available for adding attributes to Markdown elements. This is useful for fragments, amongst other things.
{{%/notice%}}
Please read the [{{%icon book%}} doc from hakimel](https://github.com/hakimel/reveal.js/#instructions)
## Options
In the frontmatter of your page file, set **type** and **revealOptions** params
Your content will be served as a fullscreen revealjs presentation and revealOptions will be used to ajust its behaviour.
+++
title = "Test slide"
type="slide"
theme = "league"
[revealOptions]
transition= 'concave'
controls= true
progress= true
history= true
center= true
+++
[read more about reveal options here](https://github.com/hakimel/reveal.js/#configuration)
## Slide Delimiters
When creating the content for your slideshow presentation within content markdown file you need to be able to distinguish between one slide and the next. This is achieved very simply using a convention within Markdown that indicates the start of each new slide.
As both horizontal and vertical slides are supported by reveal.js each has it's own unique delimiter.
To denote the start of a horizontal slide simply add the following delimiter in your Markdown:
---
To denote the start of a vertical slide simply add the following delimiter in your Markdown:
___
By using a combination of horizontal and vertical slides you can customize the navigation within your slideshow presentation. Typically vertical slides are used to present information below a top-level horizontal slide.
For example, a very simple slideshow presentation can be created as follows
```
+++
title = "test"
date = "2017-04-24T18:36:24+02:00"
type="slide"
theme = "league"
[revealOptions]
transition= 'concave'
controls= true
progress= true
history= true
center= true
+++
# In the morning
___
## Getting up
- Turn off alarm
- Get out of bed
___
## Breakfast
- Eat eggs
- Drink coffee
---
# In the evening
___
## Dinner
- Eat spaghetti
- Drink wine
___
## Going to sleep
- Get in bed
- Count sheep
```
[{{%icon expand%}}click here to view this page rendered]({{%relref "myslide.md"%}})
* [modernizr](https://modernizr.com) - A JavaScript toolkit that allows web developers to use new CSS3 and HTML5 features while maintaining a fine level of control over browsers that don't support
* [reveal-js](http://lab.hakim.se/reveal-js) - The HTML Presentation Framework
## Tooling
* [Netlify](https://www.netlify.com) - Continuous deployement and hosting of this documentation
When building the website, you can set a theme by using `--theme` option. We suggest you to edit your configuration file and set the theme by default. Example with `config.toml` format.
<!--more-->
```
theme = "docdock"
```
## Search index generation
Add the follow line in the same `config.toml` file.
```
[outputs]
home = [ "HTML", "RSS", "JSON"]
```
LUNRJS search index file will be generated on content changes.
## Your website's content
Find out how to [create]({{%relref "create-page/_index.md"%}}) and [organize your content]({{%relref "content-organisation/_index.md"%}}) quickly and intuitively.
{{% alert theme="warning" %}}HUGO **v0.25** minimum required to use this theme{{%/alert%}}
The following steps are here to help you initialize your new website. If you don’t know Hugo at all, we strongly suggest you to train by following this [great documentation for beginners](https://gohugo.io/overview/quickstart/).
<!--more-->
## Create Your Documentation
Hugo provides a `new` command to create a new website.
$ hugo new site <new_website>
## Install The Theme
Install the **Hugo-theme-docdock** theme by following this
Switch into the themes directory and download the theme
Alternatively, you can [{{%icon download%}} download the theme as .zip](https://github.com/vjeantet/hugo-theme-docdock/archive/master.zip) file and extract it in the themes directory
Docdock theme uses the last improvement available in hugo version 20+ to generate a json index file ready to be consumed by lunr.js javascript search engine.
{{%note%}}hugo generate lunrjs index.json at the root of `public` folder. <br/>When you build the site with `hugo server`, hugo generates it internally and of course it don't show up in the filesystem{{%/note%}}
The shortcurt lists files found in a **specific folder**.
Currently, it support two implementations for pages
1. If your page is a markdown file, attachements must be place in a **folder** named like your page and ending with **.files**.
> * content
> * _index.md
> * page.files
> * attachment.pdf
> * page.md
2. If your page is a **folder**, attachements must be place in a nested **'files'** folder.
> * content
> * _index.md
> * page
> * index.md
> * files
> * attachment.pdf
That's all !
{{%alert info%}}**Tip** : Look at this documentation source code on github{{%/alert%}}
### parameters
| Parameter | Default | Description |
|:--|:--|:--|
| title | "Attachments" | List's title |
| pattern | ".*" | A regular expressions, used to filter the attachments by file name. <br/><br/>{{%alert warning%}}The **pattern** parameter value must be [regular expressions](https://en.wikipedia.org/wiki/Regular_expression).
For example:
* To match a file suffix of 'jpg', use **.*jpg** (not *.jpg).
* To match file names ending in 'jpg' or 'png', use **.*(jpg|png)**
Use the children shortcode to list the child pages of a page and the further descendants (children's children). By default, the shortcode displays links to the child pages.
## Usage
| Parameter | Default | Description |
|:--|:--|:--|
| page | _current_ | Specify the page name (section name) to display children for |
| style | "li" | Choose the style used to display descendants. It could be any HTML tag name |
| showhidden | "false" | When true, child pages hidden from the menu will be displayed |
| description | "false" | Allows you to include a short text under each page in the list.<br/>when no description exists for the page, children shortcode takes the first 70 words of your content. [read more info about summaries on gohugo.io](https://gohugo.io/content/summaries/) |
| depth | 1 | Enter a number to specify the depth of descendants to display. For example, if the value is 2, the shortcode will display 2 levels of child pages. {{%alert success%}}**Tips:** set 999 to get all descendants{{%/alert%}}|
| sort | none | Sort Children By<br><li><strong>Weight</strong> - to sort on menu order</li><li><strong>Name</strong> - to sort alphabetically on menu label</li><li><strong>Identifier</strong> - to sort alphabetically on identifier set in frontmatter</li><li><strong>URL</strong> - URL</li> |
## Demo
{{%/* children */%}}
{{%children %}}
{{%/* children description="true" */%}}
{{%children description="true" %}}
{{%/* children depth="3" showhidden="true" */%}}
{{% children depth="3" showhidden="true" %}}
{{%/* children style="h2" depth="3" description="true" */%}}
{{% children style="h2" depth="3" description="true" %}}
The Excerpt Include shortcode is used to display 'excerpted' (that is, a segment of) content from one page in another.
Before you can use this shortcode, the excerpt must have been defined using the Excerpt shortcode. {{%alert info%}}Note that you can have more than one Excerpt Include shortcode on a page (although you can have only one Excerpt shortcode on a page).{{%/alert%}}
## Usage
| Parameter | Default | Description |
|:--|:--|:--|
| filename | **required** | Type the filename of the page that contains the excerpt to be displayed.<br/>Path is relative to the content folder|
| panel | none | Determines whether docDock will display a panel around the excerpted content. The panel includes the given panel's value and the border of the panel. By default, the panel and title are not shown.|
## Demo
The paragraph below shows an example of an Excerpt Include shortcode, containing content from an excerpt which we have defined on the Excerpt shortcode page. On the Excerpt Include shortcode below, we have set the options to show both the title of the page and the panel surrounding the content.
description = "The Excerpt shortcode is used to mark a part of a page's content for re-use."
+++
The Excerpt shortcode is used to mark a part of a page's content for re-use. Defining an excerpt enables other shortcodes, such as the excerpt-include shortcode, to display the marked content elsewhere.
{{%alert warning%}}You can only define one excerpt per page. In other words, you can only add the Excerpt shortcode once to a page.{{%/alert%}}
## Usage
| Parameter | Default | Description |
|:--|:--|:--|
| hidden | "false" | Controls whether the page content contained in the Excerpt shortcode placeholder is displayed on the page.{{%alert warning%}}Note that this option affects only the page that contains the Excerpt shortcode. It does not affect any pages where the content is reused.{{%/alert%}} |
## Demo
{{%/*excerpt*/%}}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation **ullamco** laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in _reprehenderit in voluptate_
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
{{%/* /excerpt*/%}}
{{%excerpt%}}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation **ullamco** laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in _reprehenderit in voluptate_
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
{{% /excerpt%}}
{{%alert info%}}See re use example with [excerpt-include shortcode]({{%relref "shortcodes/excerpt-include.md"%}}){{%/alert%}}
description = "Allow you to highlight information or put it in a box."
+++
{{% panel theme="success" header="The panel shortcode" %}}Allow you to highlight information or put it in a box. They create a colored box surrounding your text{{% /panel %}}
## Usage
| Parameter | Default | Description |
|:--|:--|:--|
| header | none | The title of the panel. If specified, this title will be displayed in its own header row. |
| footer | none | the footer of the panel. If specified, this text will be displayed in its own row |
{{%/* panel */%}}this is a panel text{{%/* /panel */%}}
{{%panel%}}this is a panel text{{%/panel%}}
## Panel with heading
Easily add a heading container to your panel with `header` parameter. You may apply any theme.
{{%/* panel theme="danger" header="panel title" */%}}this is a panel text{{%/* /panel */%}}
{{% panel theme="danger" header="panel title" %}}this is a panel text{{% /panel %}}
{{%/* panel theme="success" header="panel title" */%}}this is a panel text{{%/* /panel */%}}
{{% panel theme="success" header="panel title" %}}this is a panel text{{% /panel %}}
## Panel with footer
Wrap a secondary text in footer.
{{%/* panel footer="panel footer" */%}}Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.{{%/* /panel */%}}
{{% panel footer="panel footer" %}}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
{{% /panel %}}
## Themes
{{% panel theme="success" header="Success theme" %}}this is a panel text{{% /panel %}}
{{% panel theme="default" header="default theme" %}}this is a panel text{{% /panel %}}
{{% panel theme="primary" header="primary theme" %}}this is a panel text{{% /panel %}}
{{% panel theme="info" header="info theme" %}}this is a panel text{{% /panel %}}
{{% panel theme="warning" header="warning theme" %}}this is a panel text{{% /panel %}}
{{% panel theme="danger" header="danger theme" %}}this is a panel text{{% /panel %}}
description = "present content as a reveal.js slide"
+++
This shortcode will format the enclosed markdow to render it with [reveal.js](http://lab.hakim.se/reveal-js/) at runtime (client-side)
Read more on [revealjs github repo](https://github.com/hakimel/reveal.js/#markdown).
## Usage
`revealjs` can use the following named parameters :
* theme
* transition
* controls
* progress
* history
* center
{{%warning title="Important" %}}Even if the enclosed content is a mardown, use `<` shortcode notation instead of the `%` notation {{%/warning %}}
### Content formating and slide delimiters
[read more on this here]({{% relref "page-slide.md"%}})
## Demo
{{<revealjs theme="moon" progress="true">}}
# In the morning
___
## Getting up
- Turn off alarm
- Get out of bed
___
## Breakfast
- Eat eggs
- Drink coffee
---
# In the evening
___
## Dinner
- Eat spaghetti
- Drink wine
___
## Going to sleep
- Get in bed
- Count sheep
{{</revealjs>}}
## Source :
* [{{%icon "sunglasses" %}} click here to view raw content](https://raw.githubusercontent.com/vjeantet/hugo-theme-docdock/master/exampleSite/content/shortcodes/revealjs.md)
<!-- Place this tag where you want the button to render. -->
<aclass="github-button"href="https://github.com/vjeantet/hugo-theme-docdock/archive/master.zip"data-icon="octicon-cloud-download"aria-label="Download vjeantet/hugo-theme-docdock on GitHub">Download</a>
<!-- Place this tag where you want the button to render. -->
<aclass="github-button"href="https://github.com/vjeantet/hugo-theme-docdock"data-icon="octicon-star"data-show-count="false"aria-label="Star vjeantet/hugo-theme-docdock on GitHub">Star</a>
<!-- Place this tag where you want the button to render. -->
<aclass="github-button"href="https://github.com/vjeantet/hugo-theme-docdock/fork"data-icon="octicon-repo-forked"data-show-count="true"aria-label="Fork vjeantet/hugo-theme-docdock on GitHub">Fork</a>
</center>
<!-- Place this tag in your head or just before your close body tag. -->
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
