- Enabled metrics in storagesvc, buildermgr and controller.
- Added a middleware in storagesvc, router, executor and controller to monitor total number of http requests, each request's duration and number of requests that are currently being served. These requests can be filtered on their path, method or statuscode.
- Removed functionCallDuration and functionCallResponseSize metrics from router.
- Removed funcAliveSummary, funcIsAlive, funcReapTime and idleTime metrics.
- Replaced function calls for collecting metrics to direct metric calls.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
Co-authored-by: Sanket Sudake <sanketsudake@gmail.com>
We should avoid tap service call to executor if service URL retrieved from executor is empty.
Added sanity checks to ensure that.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Capture important open telemetry events with span in fission
* Add context to missing HTTP calls
* Add Trace ID in logs
* capture trace id in the proxy handler function
* Always registry tracer to get traceID
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* add opentracing section and otelCollectorEndpoint
* initialize OTLP exporter
* pkg/controller: changes for context propagation
* pkg/executor: changes for context propagation
* pkg/fetcher: changes for context propagation
* pkg/router: changes for context propagation
* pkg/storagesvc: changes for context propagation
* set no default value for otel collector endpoint
* update readme and add notes to charts
* move common code to pkg/utils/otel
* adding fn and env as attributes
* don't use otelhttp transport for websocket
* URL ignore with common filter UrlsToIgnore
Note: The web socket example does not work when using OTEL HTTP. Here is an issue related to that on open-telemetry/opentelemetry-js-contrib.
Signed-off-by: Gaurav Gahlot <gauravgahlot0107@gmail.com>
Co-authored-by: Sanket Sudake <sanketsudake@gmail.com>
* Retrieve pod metrics only if metrics server is running
Currently we query pod metrics every 30 sec which floods executor logs,
added check which confirms if metrics server is running then only we start
querying pod metrics for identifying CPU utilization.
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Fixed couple of typos and misspells with Go CI
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Remove unnecessary conversions with Go CI
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Enable prefix based routing
Signed-off-by: Harsh Thakur <harshthakur9030@gmail.com>
* Optimize checking condition
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Disable few tests
* disable router modification for now
* run code generator
* Collect fission dump in CI
* Enable all tests back
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Remove unwanted code
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Add prefix support at more places and couple of todo's
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Few more changes
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Improve function trim support
* Support for prefix based urls in fission function test
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* multi route for fission function test
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Improve validations in trigger creations
* Adjust leading / in url from fission
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
* Change suburl to subpath for function test
Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
Co-authored-by: Sanket Sudake <sanketsudake@gmail.com>
Passing information of function from router to executor is more efficient than calling the K8S API. This change does that instead of passing only metadata and then executor calling the K8S API again.
A ready pod which can be specialized was fetched for every function earlier, this has been changed to a queue and cache implementation in client-go to improve performance.
Poolmanager when tested at high load had some issues and this PR fixes one set of them which were found so far.
Co-authored-by: Vishal <vishal-biyani@users.noreply.github.com>
Concurrency in the pool manager allows specializing pods concurrently based on a specified limit.
Co-authored-by: Vishal <vishal-biyani@users.noreply.github.com>
The router taps function service one by one which is inefficient and
increases the burden of executor. This PR aggregates all requests into
one to solve the problem mentioned above.
If the context of request is closed before ReverseProxy finishing writing
a huge response body to the response writer, the client will only receive
a truncated response body.
To solve this, move the context cancel after ReverseProxy finished.
1. The records are stored in redis which is not migratable to another cluster for the testing purposes.
2. Some of the requests fields are not recorded.
3. People should consider using https://github.com/buger/goreplay which is an existing mature and well-tested solution for testing purposes.
The router prints error no matter what error type it is.
It's useful for troubleshooting, however, it also prints
the context canceled error, which means that users abort
request before reply and its really normal nowadays. Also,
the router returns 502 if error is not nil and may confused
client if it's a timeout error.
To solve these problems, this PR adds an error handler to
reverse proxy to examine the return error and change the
status code when needed.
This PR removes not so useful logs and changes most of Info level
log to Debug/Error level in hot path while preserving some of them
that is helpful for troubleshooting.
