Consolidates the addition of the fetcher container to the pod into a new type FetcherConfig and takes care of serviceAccountName if not set. Also added PreStop lifecycle handler if pod set TerminationGracePeriodSeconds
The root cause of the issue was introduced by PR https://github.com/fission/fission/pull/1009/files .
To be short, even the CRD of environment was delete, it still takes time for executor (poolmgr) to destroy env pool. In our cases, the previous test creates an env and delete it when test finished, then the next one creates the same name env, but failed to create pool due to the deploy name conflict. So the executor selects the pod from the first created env pool. Then, executor starts to delete the env pool, and makes the pod state became Termination state. To fix this problem, a unique name of deployment will be returned after this PR to prevent the name conflict.
Added support for Kafka headers to be passed and retrieved from functions. The headers and supported and work only for Kafka version 0.11.0.0 and higher.
* Refactor specialization process
* Separate readiness and liveness probe to different routes
For newdeploy, readiness probe should check whether a fetcher specializes env container successfully or not. In this commit, fetcher returns the actual state of current specialization status instead of returning 200ok directly.
A simple test for Kafka integration which runs locally provided cluster has Kafka and Fission install has Kafka integratiom enabled (mqtrigger-kafka deployment present).
Java environment builder with support for Maven builds. Default build comman runs `mvn clean package` and picks up target/*with-dependencies.jar for function.
Ingress integration to allow the optional creation of ingress for a given route. The ingress controller needs to be set up by the user separately so that ingress path is accessible outside the cluster.
This feature allows creation of fission objects in different namespaces, in addition to retaining the existing behavior of creating fission objects in default namespace if user doesnt provide one.
It also removes cluster admin roles for fission-fetcher and fission-builder Service Accounts and grants them only those privileges that they need.
* Fixes the issue with fn test and adds relevant test cases, fixes#650
* Checking log size for invalid function
* Testing the invalid function logs a few time beforr quitting
* Disabling the test for now, linked to issue #653
The router's cache entry for a function might become stale if the pod that had the function specialized gets deleted somehow. In such a case, we'd retry getting a new service for the function from executor and retry forwarding the user request to the newly created service.
Tests for updates to a function of new deployment executor type. Tests check for changes in the environment, scale, secrets etc. Also checks conversion of function executor type from new deployment to pool manager and vice versa
Environment Specs so far had only an image URL to specify a container image.
This was fine for public images but fell short in a few of cases:
(a) Using private image registries
(b) Specifying environment variables (this is needed for workflows helm install)
(c) Setting a SecurityContext for the container
This change adds the Container object to both build and runtime Environments.
Compatibility is preserved -- the existing ImageURL field is still used. See the comments in types.go for the overriding rules in the case that both Container and ImageURL are specified.
Newdeploy manager now checks the existence of function service cache by function UID before trying to create a new deployment. And return the cached fsvc directly if the cache exists.
setting serviceType to ClusterIP as default for fission controller and corresponding changes in fission cli to be able to port-forward the controller pod.
This change enables users to have declarative specifications for Fission resources. Users can specify their "app" in a set of spec files, and use a new fission CLI command to "apply" these specs to a running cluster.
The new "spec" CLI also includes archiving of local source files, and a file watcher that re-builds archives and uploads them on file changes, and a package build watcher that waits for package builds on the CLI.
A "--spec" option is also added to "function create", and will be added to other resources in future changes. This option causes a YAML to be outputted to the specs directory instead of the resource being created on the cluster.
The CLI "fission spec --help" outputs usage information.
This commit solves part of the issue #52 , functions are able to access secrets/configmaps specified by the user. For now, CLI only accept one secret/configmap. For advanced users, it will be able to use YAML to declare multiple secrets/configmaps in later changes.
A newdeploy backend which uses new deployment to serve requests. This is the second phase of #193 and builds on top of changes in #384 .
* Executor layer added on top of pool manager
* Removed the external server for executor
* Minor changes to keep existing semantics as much possible
* Separating the executor vs. poolmgr backend functionality and associated data members
* Executor logic separated from Poolmgr backend completely, placeholder for new backend
* Changed references to poolmgr in tests
* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package
* Rebased from master and changed references to tpr -> crd
* Executor layer added on top of pool manager
* Executor logic separated from Poolmgr backend completely, placeholder for new backend
* Changed podName to a generic objectReference in fscache (#391)
Changed podName to a generic objectReference in function service cache implementation.
* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package
* Rebased from master and changed references to tpr -> crd
* Merged from master with latest changes
* Executor layer added on top of pool manager
* Removed the external server for executor
* Minor changes to keep existing semantics as much possible
* Separating the executor vs. poolmgr backend functionality and associated data members
* Executor logic separated from Poolmgr backend completely, placeholder for new backend
* Changed references to poolmgr in tests
* update compiling.md to use helm
* Compile instructions: changed pullPolicy to IfNotPresent (#378)
Containers will get stuck in ErrImagePull/ImagePullBackOff state otherwise
* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package
* Fetcher called when pod is created for newDeploy backend but also supports older way, this is WIP and still needs pod specialization and creating & exposing a service so the URL can be hit by end user
* WIP Specializing the POD as part of startup along with fetching
* Working specialization of a new deployment. Needs some work on caching, cleanup etc.
* Switched to service based address instead of POD address
* Minor formating issue fixed
* Added logging to pods and a readiness check, the readiness check is flaky though ATM
* Fixed some rebase issues that were failing build
* Better names for K8S objects and methods
* Switched usage of FuncSvc in backends from pod to api.ObjectReference
* Adding retry to fetcher request, for now just using default retry client which might need tweaking in future
* Switching to plain old retry, some issue in getting retryablehttp with glide import
* Removed stale executor service & deployment from previous merge
* Addressed review comments, still testing some areas
* Added types in FunctionSpec
* Resolved conflicts due to merge from executor_abstraction branch
* Added backend type on EnvironmentSpec along with operations for create/list/update, the pools are created/destroyed based on change in backend type
* Backend from types and a minor err return issue fixed
* Draft version of CPU and memory parameters added to environment
* Added resourceReq to newDeploy, though it has some issues
* Issue with resourceName fixed, now newdeploy pods also pick up resources from the environment config
* Adding scale params, removing validation on CPU params for now
* Fixed a formatting issue
* Checking if slight more delay helps in the test which is currently failing for internal routes
* The resourceList newly added in Env can not be compared by compiler, hence must use breakdown comparison instead
* Added strategy selection on client side
* Added caching, informers, delete operations for newdeploy backend functions
* Deleted a stale directory
* A simple HPA based on scale parameters, testing still WIP
* Fixed a small issue in delete function, added HPA delete too when deleting a function
* Previous merge missed the pkg flag for update fn command somehow, fixed that
* Fixed comments from review
* Changed poolmgr cleanup to be generic cleanup and moved to executor, added instanceID labels to newdeploy so that cleanup works
* Moved instanceIdLabel to types to avoid cyclic dependency
* More review fixes
* Tweaking sleep to see results
* If user does not provide poolsize, then it should not default to zero
* Switched to naming convention for now, fixed default poolsize if not provided
* Changed error return behaviour in delete fn, also changed cleanup to look based on obj type though support for additional type will need more work
* Changed check location so avoid false logging
* Test for newdeploy backend
* Adding tests for poolmgr backend
* Fixed an issue with glide dependency version, already fixed in master
* Added instanceId for NewDeploy, Initial cleanup now cleans older objects of newdeploy backend, removed eagercreate flag and instead using minScale to drive eager creation
* Moved cleanup to executor layer with cleanup for newDeploy backend, changes to use the new Cache impl
* Cleaning up pod & rs along with deployment for newdeploy backend
* Enhanced fn and env listing to show min/maxscale and resuorces respectively
* Added conditional heapster deployment and fixed a small issue with resources for fetcher container in function pod
* Addressed review comments from previous change
* Addressed some more review comments - majorly create only on NotFoundError
* Added TargetCPU as an input for scaling
* Bumped target CPU to be greater than 0 and added a default value
* Min replicas should be 1 even if the minScale is 0 when creating deployment
* Changed name from 'backend' to executorType, added additional test for minscale 0 case, changed TargetCPU to TargetCPUPercent
All functions have a pkg reference. This can be a package with either source and a deploy archives, or, a deploy archive. Everytime a function is updated, a new package is created. With archive pruner, the archives that are pointed to by old pkg reference can be deleted from the storage.
* High level spec for package pruning.
* Skeleton for archive pruning
* Adding meat 1 to skeleton.
* Adding meat #2. Separated storage service into a httpHandler component and
Storage Layer component.
* Adding meat #3. getOrphanedArchives in pruner and getItems on
stowClient.
* Restructured archivePruner methods.
* Commiting the day's work. Ready for testing #1.
* Fixing compile errors.
* Test ready. added a few logs for debugging.
* Adding a filter for getItems in stowClient.
* After testing.
* Added a test for archivePruner.
* Adding helm value pruneInterval for testing.
* Modified test.
* Final test.
* Fixing interval from seconds to minutes.
* Small change.
* Changing debugs to info.
* Removing the WIP design
* Ran gofmt on all these files.
* Fixing prune_interval as string in ENV var.
* Addressing all comments, but one.
* changing getFile method in stowClient to stream it into a response.
* All comments incorporated.
* Introducing a new flag for running archivePruner.
1. This flag is disabled for archivePruner to run in unit test.
2. This flag is enabled for archivePruner to run in production.
3. Also disabling test_archive_pruner.sh in this PR. Follow up with
next PR to enable it.
* Addressing review comments.
* Changing the command to generate a file dynamically.
* Enabling arching_pruner_test
* giving execute permissions to test_archive_pruner.sh
* Making changes of positional parameters after recent commit.
Change test case permission and removing kubectlPortForward.
* Adding debug to see why test_utils.sh passed junk pruneInterval.
* shell needs special handling for positional parameters from 10.
