Files
fission-src/types.go
T
VishalandTa-Ching Chen 4cf195768e Newdeploy backend (#387)
A newdeploy backend which uses new deployment to serve requests. This is the second phase of #193 and builds on top of changes in #384 .

* Executor layer added on top of pool manager

* Removed the external server for executor

* Minor changes to keep existing semantics as much possible

* Separating the executor vs. poolmgr backend functionality and associated data members

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed references to poolmgr in tests

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Rebased from master and changed references to tpr -> crd

* Executor layer added on top of pool manager

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed podName to a generic objectReference in fscache (#391)

Changed podName to a generic objectReference in function service cache implementation.

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Rebased from master and changed references to tpr -> crd

* Merged from master with latest changes

* Executor layer added on top of pool manager

* Removed the external server for executor

* Minor changes to keep existing semantics as much possible

* Separating the executor vs. poolmgr backend functionality and associated data members

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed references to poolmgr in tests

* update compiling.md to use helm

* Compile instructions: changed pullPolicy to IfNotPresent (#378)

Containers will get stuck in ErrImagePull/ImagePullBackOff state otherwise

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Fetcher called when pod is created for newDeploy backend but also supports older way, this is WIP and still needs pod specialization and creating & exposing a service so the URL can be hit by end user

* WIP Specializing the POD as part of startup along with fetching

* Working specialization of a new deployment. Needs some work on caching, cleanup etc.

* Switched to service based address instead of POD address

* Minor formating issue fixed

* Added logging to pods and a readiness check, the readiness check is flaky though ATM

* Fixed some rebase issues that were failing build

* Better names for K8S objects and methods

* Switched usage of FuncSvc in backends from pod to api.ObjectReference

* Adding retry to fetcher request, for now just using default retry client which might need tweaking in future

* Switching to plain old retry, some issue in getting retryablehttp with glide import

* Removed stale executor service & deployment from previous merge

* Addressed review comments, still testing some areas

* Added types in FunctionSpec

* Resolved conflicts due to merge from executor_abstraction branch

* Added backend type on EnvironmentSpec along with operations for create/list/update, the pools are created/destroyed based on change in backend type

* Backend from types and a minor err return issue fixed

* Draft version of CPU and memory parameters added to environment

* Added resourceReq to newDeploy, though it has some issues

* Issue with resourceName fixed, now newdeploy pods also pick up resources from the environment config

* Adding scale params, removing validation on CPU params for now

* Fixed a formatting issue

* Checking if slight more delay helps in the test which is currently failing for internal routes

* The resourceList newly added in Env can not be compared by compiler, hence must use breakdown comparison instead

* Added strategy selection on client side

* Added caching, informers, delete operations for newdeploy backend functions

* Deleted a stale directory

* A simple HPA based on scale parameters, testing still WIP

* Fixed a small issue in delete function, added HPA delete too when deleting a function

* Previous merge missed the pkg flag for update fn command somehow, fixed that

* Fixed comments from review

* Changed poolmgr cleanup to be generic cleanup and moved to executor, added instanceID labels to newdeploy so that cleanup works

* Moved instanceIdLabel to types to avoid cyclic dependency

* More review fixes

* Tweaking sleep to see results

* If user does not provide poolsize, then it should not default to zero

* Switched to naming convention for now, fixed default poolsize if not provided

* Changed error return behaviour in delete fn, also changed cleanup to look based on obj type though support for additional type will need more work

* Changed check location so avoid false logging

* Test for newdeploy backend

* Adding tests for poolmgr backend

* Fixed an issue with glide dependency version, already fixed in master

* Added instanceId for NewDeploy, Initial cleanup now cleans older objects of newdeploy backend, removed eagercreate flag and instead using minScale to drive eager creation

* Moved cleanup to executor layer with cleanup for newDeploy backend, changes to use the new Cache impl

* Cleaning up pod & rs along with deployment for newdeploy backend

* Enhanced fn and env listing to show min/maxscale and resuorces respectively

* Added conditional heapster deployment and fixed a small issue with resources for fetcher container in function pod

* Addressed review comments from previous change

* Addressed some more review comments - majorly create only on NotFoundError

* Added TargetCPU as an input for scaling

* Bumped target CPU to be greater than 0 and added a default value

* Min replicas should be 1 even if the minScale is 0 when creating deployment

* Changed name from 'backend' to executorType, added additional test for minscale 0 case, changed TargetCPU to TargetCPUPercent
2018-02-03 01:02:28 +08:00

381 lines
11 KiB
Go

/*
Copyright 2016 The Fission Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package fission
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/pkg/api/v1"
)
type (
//
// Functions and packages
//
// ChecksumType specifies the checksum algorithm, such as
// sha256, used for a checksum.
ChecksumType string
// Checksum of package contents when the contents are stored
// outside the Package struct. Type is the checksum algorithm;
// "sha256" is the only currently supported one. Sum is hex
// encoded.
Checksum struct {
Type ChecksumType `json:"type"`
Sum string `json:"sum"`
}
// ArchiveType is either literal or URL, indicating whether
// the package is specified in the Archive struct or
// externally.
ArchiveType string
// Package contains or references a collection of source or
// binary files.
Archive struct {
// Type defines how the package is specified: literal or URL.
Type ArchiveType `json:"type"`
// Literal contents of the package. Can be used for
// encoding packages below TODO (256KB?) size.
Literal []byte `json:"literal"`
// URL references a package.
URL string `json:"url"`
// Checksum ensures the integrity of packages
// refereced by URL. Ignored for literals.
Checksum Checksum `json:"checksum"`
}
EnvironmentReference struct {
Namespace string `json:"namespace"`
Name string `json:"name"`
}
BuildStatus string
PackageSpec struct {
Environment EnvironmentReference `json:"environment"`
Source Archive `json:"source"`
Deployment Archive `json:"deployment"`
BuildCommand string `json:"buildcmd"`
// In the future, we can have a debug build here too
}
PackageStatus struct {
BuildStatus BuildStatus `json:"buildstatus"`
BuildLog string `json:"buildlog"` // output of the build (errors etc)
}
PackageRef struct {
Namespace string `json:"namespace"`
Name string `json:"name"`
// Including resource version in the reference forces the function to be updated on
// package update, making it possible to cache the function based on its metadata.
ResourceVersion string `json:"resourceversion"`
}
FunctionPackageRef struct {
PackageRef PackageRef `json:"packageref"`
// FunctionName specifies a specific function within the package. This allows
// functions to share packages, by having different functions within the same
// package.
//
// Fission itself does not interpret this path. It is passed verbatim to
// build and runtime environments.
//
// This is optional: if unspecified, the environment has a default name.
FunctionName string `json:"functionName"`
}
//ExecutorType is the primary executor for an environment
ExecutorType string
//StrategyType is the strategy to be used for function execution
StrategyType string
// FunctionSpec describes the contents of the function.
FunctionSpec struct {
// Environment is the build and runtime environment that this function is
// associated with. An Environment with this name should exist, otherwise the
// function cannot be invoked.
Environment EnvironmentReference `json:"environment"`
// Reference to a package containing deployment and optionally the source
Package FunctionPackageRef `json:"package"`
// cpu and memory resources as per K8S standards
Resources v1.ResourceRequirements `json:"resources"`
// InvokeStrategy is a set of controls which affect how function executes
InvokeStrategy InvokeStrategy
}
/*InvokeStrategy is a set of controls over how the function executes.
It affects the performance and resource usage of the function.
An InvokeStategy is of one of two types: ExecutionStrategy, which controls low-level
parameters such as which ExecutorType to use, when to autoscale, minimum and maximum
number of running instances, etc. A higher-level AbstractInvokeStrategy will also be
supported; this strategy would specify the target request rate of the function,
the target latency statistics, and the target cost (in terms of compute resources).
*/
InvokeStrategy struct {
ExecutionStrategy ExecutionStrategy
StrategyType StrategyType
}
/*ExecutionStrategy specifies low-level parameters for function execution,
such as the number of instances.
MinScale affects the cold start behaviour for a function. If MinScale is 0 then the
deployment is created on first invocation of function and is good for requests of
asynchronous nature. If MinScale is greater than 0 then MinScale number of pods are
created at the time of creation of function. This ensures faster response during first
invocation at the cost of consuming resources.
MaxScale is the maximum number of pods that function will scale to based on TargetCPUPercent
and resources allocated to the function pod.
*/
ExecutionStrategy struct {
ExecutorType ExecutorType
MinScale int
MaxScale int
TargetCPUPercent int
}
FunctionReferenceType string
FunctionReference struct {
// Type indicates whether this function reference is by name or selector. For now,
// the only supported reference type is by name. Future reference types:
// * Function by label or annotation
// * Branch or tag of a versioned function
// * A "rolling upgrade" from one version of a function to another
Type FunctionReferenceType `json:"type"`
// Name of the function.
Name string `json:"name"`
}
//
// Environments
//
Runtime struct {
// Image for containing the language runtime.
Image string `json:"image"`
// LoadEndpointPort defines the port on which the
// server listens for function load
// requests. Optional; default 8888.
LoadEndpointPort int32 `json:"loadendpointport"`
// LoadEndpointPath defines the relative URL on which
// the server listens for function load
// requests. Optional; default "/specialize".
LoadEndpointPath string `json:"loadendpointpath"`
// FunctionEndpointPort defines the port on which the
// server listens for function requests. Optional;
// default 8888.
FunctionEndpointPort int32 `json:"functionendpointport"`
}
Builder struct {
// Image for containing the language runtime.
Image string `json:"image"`
// (Optional) Default build command to run for this build environment.
Command string `json:"command"`
}
EnvironmentSpec struct {
// Environment API version
Version int `json:"version"`
// Runtime container image etc.; required
Runtime Runtime `json:"runtime"`
// Optional
Builder Builder `json:"builder"`
// Optional, but strongly encouraged. Used to populate
// links from UI, CLI, etc.
DocumentationURL string `json:"documentationurl"`
// Optional
// Defaults to 'Single'
AllowedFunctionsPerContainer AllowedFunctionsPerContainer `json:"allowedFunctionsPerContainer"`
// Request and limit resources for the environment
Resources v1.ResourceRequirements `json:"resources"`
// The initial pool size for environment
Poolsize int `json:"poolsize"`
}
AllowedFunctionsPerContainer string
//
// Triggers
//
HTTPTriggerSpec struct {
Host string `json:"host"`
RelativeURL string `json:"relativeurl"`
Method string `json:"method"`
FunctionReference FunctionReference `json:"functionref"`
}
KubernetesWatchTriggerSpec struct {
Namespace string `json:"namespace"`
Type string `json:"type"`
LabelSelector map[string]string `json:"labelselector"`
FunctionReference FunctionReference `json:"functionref"`
}
// MessageQueueTriggerSpec defines a binding from a topic in a
// message queue to a function.
MessageQueueTriggerSpec struct {
FunctionReference FunctionReference `json:"functionref"`
MessageQueueType string `json:"messageQueueType"`
Topic string `json:"topic"`
ResponseTopic string `json:"respTopic,omitempty"`
ContentType string `json:"contentType"`
}
// TimeTrigger invokes the specific function at a time or
// times specified by a cron string.
TimeTriggerSpec struct {
Cron string `json:"cron"`
FunctionReference `json:"functionref"`
}
// Errors returned by the Fission API.
Error struct {
Code errorCode `json:"code"`
Message string `json:"message"`
}
errorCode int
)
//
// Fission-Environment interface. The following types are not
// exposed in the Fission API, but rather used by Fission to
// talk to environments.
//
type (
FunctionLoadRequest struct {
// FilePath is an absolute filesystem path to the
// function. What exactly is stored here is
// env-specific. Optional.
FilePath string `json:"filepath"`
// FunctionName has an environment-specific meaning;
// usually, it defines a function within a module
// containing multiple functions. Optional; default is
// environment-specific.
FunctionName string `json:"functionName"`
// URL to expose this function at. Optional; defaults
// to "/".
URL string `json:"url"`
// Metatdata
FunctionMetadata *metav1.ObjectMeta
}
)
const EXECUTOR_INSTANCEID_LABEL string = "executorInstanceId"
const (
ChecksumTypeSHA256 ChecksumType = "sha256"
)
const (
// ArchiveTypeLiteral means the package contents are specified in the Literal field of
// resource itself.
ArchiveTypeLiteral ArchiveType = "literal"
// ArchiveTypeUrl means the package contents are at the specified URL.
ArchiveTypeUrl ArchiveType = "url"
)
const (
BuildStatusPending = "pending"
BuildStatusRunning = "running"
BuildStatusSucceeded = "succeeded"
BuildStatusFailed = "failed"
BuildStatusNone = "none"
)
const (
AllowedFunctionsPerContainerSingle = "single"
AllowedFunctionsPerContainerInfinite = "infinite"
)
const (
ExecutorTypePoolmgr = "poolmgr"
ExecutorTypeNewdeploy = "newdeploy"
)
const (
StrategyTypeExecution = "execution"
)
const (
// FunctionReferenceFunctionName means that the function
// reference is simply by function name.
FunctionReferenceTypeFunctionName = "name"
// Other function reference types we'd like to support:
// Versioned function, latest version
// Versioned function. by semver "latest compatible"
// Set of function references (recursively), by percentage of traffic
)
const (
ErrorInternal = iota
ErrorNotAuthorized
ErrorNotFound
ErrorNameExists
ErrorInvalidArgument
ErrorNoSpace
ErrorNotImplmented
ErrorChecksumFail
ErrorSizeLimitExceeded
)
// must match order and len of the above const
var errorDescriptions = []string{
"Internal error",
"Not authorized",
"Resource not found",
"Resource exists",
"Invalid argument",
"No space",
"Not implemented",
"Checksum verification failed",
"Size limit exceeded",
}
const (
ArchiveLiteralSizeLimit int64 = 256 * 1024
)