Remove the `logger` container from the logging daemonset.
Remove the outgoing call from the poolmgr to the logger. Use Fluentd's Kubernetes filter to add function name and UID to influx metadata.
This means fluentd now figures out when to start collecting function logs on its own, without being informed by poolmgr. This is great for other execution strategies, and for autoscaling, where fission isn't in direct control of function pod creation.
Also adds an integration test to make sure logging keeps working.
This contains a few different builder manager fixes:
* Update the zip file structure to avoid an extra subdirectory
* Annotate packages with what functions are using them
* Only trigger builds when there is no deployment archive
* Python environment loadpath bugfixes
* Other bugfixes
Switch Fission's storage over to the new CustomResourceDefinitions, from the deprecated ThirdPartyResources. This allows us to be compatible with Kubernets 1.8 and onwards.
This also adds a CLI tool for dumping state from an old fission version and restoring state into new CRDs.
The storage service is unaffected by this change.
Add a flag to the environment to control multiple specialization -- the max number of functions per container. This can be set to 1 or infinity.
Add an api proxy to workflow apiserver from the controller.
Add function metadata to FunctionLoadRequest; every v2 environment now knows which function it's loading (but can ignore that information if it wants to).
Add function identity headers to router. This is useful for multiple specialization, so the router can disambiguate between different function calls. (If this turns out to be a non-trivial perf overhead, we could add these headers conditionally, but for now they are always added.)
This change orchestrates function builds.
Environments (in v2) define a builder image, just like they do a runtime image. The builder image contains a build script that's invoked with source and deployment paths (as env vars).
The buildermgr watches for environments with build images defined, and creates build deployments and services.
Functions can define source and deployment. Buildermgr watches for functions with source code (and build status == pending) and invokes the environment's builder when appropriate. It captures logs from the build and sets the build status (success/failure) and build lots into the PackageStatus.
This adds partial support for multiple specialization to the
function service cache. It allows Add() to succeed if the pod and
address maps already contain entries.
But it doesn't deal with DeletePod and TouchByAddress. That's ok for
now, because the workflow engine never deletes the environment pod
(that pod is the workflow engine).
However, if/when we want full support for multiple specialization,
that will require updating both DeletePod and TouchByAddress.
Introduce _Archives_ as a type to reference arbitrary blobs.
_Packages_ are a pair of Source and Deployment archives. Packages have an environment reference.
Functions reference a package. Multiple functions can reference the same package.
Watch functions in the router and use that to trigger functionReferenceResolver cache invalidation.
We might want to rate-limit syncTriggers in a future change, since we might be triggering it more often than needed.
Split out the Package type into a first class Kubernetes resource. Before this change, packages were implicitly tied to functions.
This wasn't ideal because:
* Functions will need to share packages
* A package storage system may be more generally useful than just
functions (for example, for storing static assets)
This change does the following:
* Updates the fission and tpr types to add a new Package and PackageSpec. It also creates a PackageRef type, and a FunctionPackageRef type. The PackageRef simply references a package, but the FunctionPackageRef includes the name of a function within the package. This allows us to share packages between different functions.
* Updates fetcher and other components for first-class packages
* Allows customization of fetcher image pull policy in the helm charts
This was initially added to avoid restarts if poolmgr starts before
fission api; but we don't really need that since k8s will restart
it. Also, poolmgr now talks to the k8s api directly instead of the
fission api.
This changes the core fission function, environment and trigger types. It also changes Fission's storage to use ThirdPartyResources.
- Functions are now specified by packages. Functions can also have both source and deployment packages. A package can be specified by a literal, or by a URL.
- Environments have a build and runtime component.
- Triggers reference functions by a FunctionReference. This is a layer of indirection between triggers and functions, and will allow things like incremental function upgrades in future releases.
See Documentation/wip/env-v2.md for design discussion about points 1 and 2.
Changes:
* V2 Types
All types now have a spec, following the pattern of K8s objects.
Functions now have source and deployment packages. A Package can be
specified by literal, or by URL.
Environments now have a builder and runtime component.
All triggers use a new FunctionReference to specify the function. This
for now only uses a function name, but in the future can be extended
to be more flexible.
A new FunctionLoadRequest type is added for specialization requests to
the environment runtime.
* TPR types, TPR init code, and a "fission client"
Implements TPR types using the spec types in fission/types.go.
Adds code for adding creating TPR types, and convenient types for crud
operations on each of our resource types.
Adds code for connecting to K8s API and configuring a REST client with
fission types set up.
* Change old stateful controller into a thin apiserver
This apiserver is now simply a stateless api layer on top of the TPR
types. At the moment it doesn't do anything that couldn't be done by
simply talking to the TPR types. In the future we can have better
validation and potentially some higher level APIs (like versioning for
example) in here.
* Split controller client into files and update for v2 types.
* Update CLI for v2 types.
As far as possible we keep the CLI flags the same. We'll have to add
flags for source/deploy packages and builder/runtime
environments. That will come in the next change.
* Update poolmgr and fetcher for v2 types.
Also adds a poolmgr_test.
* Update router for new types.
Also adds a function reference resolver, which separates out the job
of resolving a FunctionReference to a function.
* Update kubewatcher and timer for v2 types.
* Update Message Queue trigger type for v2 types.
* Minor odds and ends.
* Fission bundle CLI updates
Remove controllerUrl flag, since we don't need it any more.
* Remove etcd deployment (replaced by storing state in TPR)
Also update the poolmgr commandline, and use an env var for the
fetcher image URL.
* Explicit ChecksumType and consts
* Clarify separation of environment interface types
The poolmgr client now aggregates successive tapService() calls and sends them to the poolmgr service every 5 sec, instead of sending them immediately.
* Fix controller timer handler not return when error occurred
* Fix controller not close http response correctly
This PR aims to fix controller not close response correctly. Also, I’ve check all over the project there is no more resource leak issues exist after this patch.
* Fix controller not return immediately when error occurred
* Sort import
The generic pool manager creates a generic pool with the reference of environment metadata. Since the reference is address of loop variable, so the content of metadata will be replaced with the last element of envs. So pool manager sets up logging with wrong env.
Add function log aggregation and persistence using Fluentd and InfluxDB.
Fluentd and a helper sidecar run as a daemonset. The poolmgr sets up logging for each function pod, using the helper sidecar. Fluentd forwards logs to InfluxDB, which is run as a deployment and service. The client CLI directly queries InfluxDB for logs.
Fluentd supports many outputs besides InfluxDB, so we aren't very tied to InfluxDB.
The setup is somewhat manual, which we should be able to improve by integrating this into the helm chart.
Diagram of component interactions: https://cloud.githubusercontent.com/assets/202578/23100399/b0e3ea00-f6ba-11e6-8f2f-6588cfef2e84.png
Fixes#139. The code before this change was saving up pointers to the loop variable in a slice. So loop was useless and we always chose the last pod in the list, whether it was ready or not. With this fix, choosePod should always return a ready pod.
Also adds a check for the pod phase, before looping over the containers in the pod, and discard pods that aren't running yet.
Addresses issue #113. poolmgr.MakeGenericPool does not need to wait
for a ready pod. Kubernetes already retries image fetching, so
there's no need to repeat the task in fission. And if it's a
non-retriable error (like the image URL being wrong), then the
deployment will stay broken, and the user can fix the environment with
'fission env update' (or 'fission env delete' followed by 'fission env
create').
Label all poolmgr-created resources with an id that's unique to a
running poolmgr instance. On poolmgr start up, clean up resources
created by old instances. Resources that are idle are killed
immediately; resources that could be running a user function are
killed after the maximum function timeout.
Addresses issue #51. Makes versioning in poolmgr explicit, so its
cache needs no invalidation on version update. Functions in poolmgr
are always cached by name and UID.
Router now updates implictly versioned routes with the latest version
of a function.
This means that users will see requests to implicitly-versioned routes
go to the latest version of a function within 3 seconds. (Those 3 sec
will go away when we use a real watch instead of polling the
controller.)
There's no change in behaviour for routes that explictly specify a
function UID.
A defer statement contained an access to a pointer that was set to
nil before the deferred statement was run; remove it and just
explicity run the statement.
Idle pod reaper wakes up once a minute, looks in the functionServiceCache for pods that
haven't been accessed for more than idlePodReapTime, and deletes all such pods.
(This doesn't yet delete pods that may be leaked from previously terminated poolmgrs; it
only works with pods created by the current poolmgr instance.)
Move separate caches to one cache -- functionServiceCache. It can be
looked up by function, can update atime by address, and can be deleted
by podname. This removes the other caches. Some of the concurrency
logic is still a bit hairy; it might be better not to use fission.Cache.
Delete pods that are unused for more than a certain timeout. This
change isn't complete -- other funcSvc caches need to be invalidated
on delete. This needs a bit of refactoring.
Poolmgr needs to know usage statistics for a function's pod. This
change asynchronously taps poolmgr API when router uses a service.
Poolmgr can use this information to control pod expiry. It may also
be useful later as one of the metrics for autoscaling.
Even though the K8s api returns quickly after creating a service, that
service doesn't seem usage until about a second or two later. We need
to investigate this and see if there's something we can do to make it
faster. If there isn't we'll remove the svc code entirely; for now
it's behind a useSvc flag that's set to false.
If the pool is new or very busy, we may call the chosen container's
specialize endpoint before it's actually up -- handle the case by
retrying a few times.
Also namespace-qualify service hostname (since router and
functions run in different namespaces).
:= attempts to declare as many of the variables on its left side as it
can, instead of re-using as many as it can. Consider this code:
a, ok := foo()
if !ok {
a, err := bar()
...
}
The inner 'a' is a different var from the outer one, and goes out of
scope at the }. The outer 'a' is left with whatever value foo()
returned.
