- Currently, fission components don't handle shutdown signals.
So we don't get any to do the required cleanup before the fission process
exits. Adding signal capture process with cancelling context so
that all dependent processes stop working when the process gets term
signal.
- Set log level to error in otel shutdown function
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
Reactored ready pod controller code to user lister and cache sync checks,
so that we avoid querying lister if cache is not synced in choodPod function.
Also, as noticied in #2258 we were initializing workqueue in goroutine
which was causing nil pointer reference. We have moved it out of goroutine
and kept specific parts in goroutine.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Capture important open telemetry events with span in fission
* Add context to missing HTTP calls
* Add Trace ID in logs
* capture trace id in the proxy handler function
* Always registry tracer to get traceID
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
In upgrade tests, sometimes in race condition pool manager podlister
takes time to sync in which case, poolpodcontrolller get nil reference
for gpm.podLister. Passing podInformer to poolpodcontroller so that
we wait for podInformer cache to sync.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
Using controller-runtime zap integration which is aware of Kubernetesobjects and logs only name+namespace of the object when complete runtime.Object compatible type is given to logger.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
Signed-off-by: Sanket Sudake sanketsudake@gmail.com
- Use informers and listers in executors
- Passing context properly in pool manager executor
- Environment updates in the pool manager would not cause updates in the deployment
- Environment update minimizing downtime
- Use replicaset controller and environment delete triggers to cleanup specialized pods
Generally using typed informers is more standard practise
than using SharedIndexInformer(SII) implicity. SII also lack
listers provided by informer factory and few other high level
abstractions.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
Currently most of informers used in Fission are using resync
period of 30sec which is quite low and causes lot of disturbances
in overall system. Since informers already have watchers which would
be delivering events to the handlers. If any handlers require lower
sync period they should register handler with `AddEventHandlersWithResyncPeriod`
instead of we generaling informer.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* add opentracing section and otelCollectorEndpoint
* initialize OTLP exporter
* pkg/controller: changes for context propagation
* pkg/executor: changes for context propagation
* pkg/fetcher: changes for context propagation
* pkg/router: changes for context propagation
* pkg/storagesvc: changes for context propagation
* set no default value for otel collector endpoint
* update readme and add notes to charts
* move common code to pkg/utils/otel
* adding fn and env as attributes
* don't use otelhttp transport for websocket
* URL ignore with common filter UrlsToIgnore
Note: The web socket example does not work when using OTEL HTTP. Here is an issue related to that on open-telemetry/opentelemetry-js-contrib.
Signed-off-by: Gaurav Gahlot <gauravgahlot0107@gmail.com>
Co-authored-by: Sanket Sudake <sanketsudake@gmail.com>
The patch adds few improvements in pool manager and adds better
function composability by reorganizing code.
1. Added created status in get pool call
2. Improved logging in certain areas and having logger per component
3. Separated deployment-specific code in diff file for extensibility
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
Currently when we update env, even if we dont have newdeploy related
functions env is getting is getting triggered. Added filter in finding
related functions to avoid this behaviour.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
PR adds a new executortype which supports running containers as functions. New CLI under functions is added to create container as functions.
Co-authored-by: Harsh Thakur <harshthakur9030@gmail.com>
Co-authored-by: Sanket Sudake <sanketsudake@gmail.com>
1. Support for adding labels and annotations to functions & environment via fission CLI through create & update command.
2. Change ensures labels and annotations assigned to the environment would reflect on pods created via executor type pool manager and new deploy.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Retrieve pod metrics only if metrics server is running
Currently we query pod metrics every 30 sec which floods executor logs,
added check which confirms if metrics server is running then only we start
querying pod metrics for identifying CPU utilization.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Fixed couple of typos and misspells with Go CI
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Remove unnecessary conversions with Go CI
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Add a SharedIndexInformer for services and deployments to NewDeploy executor.
This brings the NewDeploy executor behaviour into sync with GenericPoolManager behaviour
by caching Kubernetes services and deployments used in per-request function validation.
* Create informers in executer by executor label
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
Co-authored-by: James Sinclair <irgeek@btc.com.au>
Adding once only execution pattern to pool manager based functions. This is for use cases where you don't want to share the samne pod for another execution instance!
Websocket event support for cleaning up pods only after WS connection is terminated. The support for websocket is right now in the environment itself and router simply acts as a proxy for WS communication!
Increase the default concurrency to 500 from 5 set in the function. Usually, it is good to error on the higher side and if needed, users can always fine-tune it to their needs.
This feature enables routing more than one request to a pod at the same time. This is the first draft of the work and might involve more optimizations later.
Passing information of function from router to executor is more efficient than calling the K8S API. This change does that instead of passing only metadata and then executor calling the K8S API again.
A ready pod which can be specialized was fetched for every function earlier, this has been changed to a queue and cache implementation in client-go to improve performance.
Poolmanager when tested at high load had some issues and this PR fixes one set of them which were found so far.
Co-authored-by: Vishal <vishal-biyani@users.noreply.github.com>
Concurrency in the pool manager allows specializing pods concurrently based on a specified limit.
Co-authored-by: Vishal <vishal-biyani@users.noreply.github.com>
Attribute enableServiceLinks can be now configured in PodSpec so that environment variables are not injected in function pods and DNS is used for discovery
Executor wrongly deletes role bindings if the user
creates an environment in the reserved namespaces. This
PR is a quick fix to solve the problem by checking if
an environment is under reserved namespaces.
The root cause of the problem is that eagerPoolCreator
tries to create the deployment when the poolmanager
is trying to delete it. To avoid this, start eager pool
creator after executor starts serving requests.
