Newdeploy backend (#387)
A newdeploy backend which uses new deployment to serve requests. This is the second phase of #193 and builds on top of changes in #384 . * Executor layer added on top of pool manager * Removed the external server for executor * Minor changes to keep existing semantics as much possible * Separating the executor vs. poolmgr backend functionality and associated data members * Executor logic separated from Poolmgr backend completely, placeholder for new backend * Changed references to poolmgr in tests * Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package * Rebased from master and changed references to tpr -> crd * Executor layer added on top of pool manager * Executor logic separated from Poolmgr backend completely, placeholder for new backend * Changed podName to a generic objectReference in fscache (#391) Changed podName to a generic objectReference in function service cache implementation. * Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package * Rebased from master and changed references to tpr -> crd * Merged from master with latest changes * Executor layer added on top of pool manager * Removed the external server for executor * Minor changes to keep existing semantics as much possible * Separating the executor vs. poolmgr backend functionality and associated data members * Executor logic separated from Poolmgr backend completely, placeholder for new backend * Changed references to poolmgr in tests * update compiling.md to use helm * Compile instructions: changed pullPolicy to IfNotPresent (#378) Containers will get stuck in ErrImagePull/ImagePullBackOff state otherwise * Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package * Fetcher called when pod is created for newDeploy backend but also supports older way, this is WIP and still needs pod specialization and creating & exposing a service so the URL can be hit by end user * WIP Specializing the POD as part of startup along with fetching * Working specialization of a new deployment. Needs some work on caching, cleanup etc. * Switched to service based address instead of POD address * Minor formating issue fixed * Added logging to pods and a readiness check, the readiness check is flaky though ATM * Fixed some rebase issues that were failing build * Better names for K8S objects and methods * Switched usage of FuncSvc in backends from pod to api.ObjectReference * Adding retry to fetcher request, for now just using default retry client which might need tweaking in future * Switching to plain old retry, some issue in getting retryablehttp with glide import * Removed stale executor service & deployment from previous merge * Addressed review comments, still testing some areas * Added types in FunctionSpec * Resolved conflicts due to merge from executor_abstraction branch * Added backend type on EnvironmentSpec along with operations for create/list/update, the pools are created/destroyed based on change in backend type * Backend from types and a minor err return issue fixed * Draft version of CPU and memory parameters added to environment * Added resourceReq to newDeploy, though it has some issues * Issue with resourceName fixed, now newdeploy pods also pick up resources from the environment config * Adding scale params, removing validation on CPU params for now * Fixed a formatting issue * Checking if slight more delay helps in the test which is currently failing for internal routes * The resourceList newly added in Env can not be compared by compiler, hence must use breakdown comparison instead * Added strategy selection on client side * Added caching, informers, delete operations for newdeploy backend functions * Deleted a stale directory * A simple HPA based on scale parameters, testing still WIP * Fixed a small issue in delete function, added HPA delete too when deleting a function * Previous merge missed the pkg flag for update fn command somehow, fixed that * Fixed comments from review * Changed poolmgr cleanup to be generic cleanup and moved to executor, added instanceID labels to newdeploy so that cleanup works * Moved instanceIdLabel to types to avoid cyclic dependency * More review fixes * Tweaking sleep to see results * If user does not provide poolsize, then it should not default to zero * Switched to naming convention for now, fixed default poolsize if not provided * Changed error return behaviour in delete fn, also changed cleanup to look based on obj type though support for additional type will need more work * Changed check location so avoid false logging * Test for newdeploy backend * Adding tests for poolmgr backend * Fixed an issue with glide dependency version, already fixed in master * Added instanceId for NewDeploy, Initial cleanup now cleans older objects of newdeploy backend, removed eagercreate flag and instead using minScale to drive eager creation * Moved cleanup to executor layer with cleanup for newDeploy backend, changes to use the new Cache impl * Cleaning up pod & rs along with deployment for newdeploy backend * Enhanced fn and env listing to show min/maxscale and resuorces respectively * Added conditional heapster deployment and fixed a small issue with resources for fetcher container in function pod * Addressed review comments from previous change * Addressed some more review comments - majorly create only on NotFoundError * Added TargetCPU as an input for scaling * Bumped target CPU to be greater than 0 and added a default value * Min replicas should be 1 even if the minScale is 0 when creating deployment * Changed name from 'backend' to executorType, added additional test for minscale 0 case, changed TargetCPU to TargetCPUPercent
This commit is contained in:
Vishal
+13
-77
@@ -47,7 +47,6 @@ import (
|
||||
"github.com/fission/fission/executor/fscache"
|
||||
)
|
||||
|
||||
const POOLMGR_INSTANCEID_LABEL string = "poolmgrInstanceId"
|
||||
const POD_PHASE_RUNNING string = "Running"
|
||||
|
||||
type (
|
||||
@@ -144,9 +143,10 @@ func MakeGenericPool(
|
||||
|
||||
// Labels for generic deployment/RS/pods.
|
||||
gp.labelsForPool = map[string]string{
|
||||
"environmentName": gp.env.Metadata.Name,
|
||||
"environmentUid": string(gp.env.Metadata.UID),
|
||||
POOLMGR_INSTANCEID_LABEL: gp.instanceId,
|
||||
"environmentName": gp.env.Metadata.Name,
|
||||
"environmentUid": string(gp.env.Metadata.UID),
|
||||
fission.EXECUTOR_INSTANCEID_LABEL: gp.instanceId,
|
||||
"executorType": fission.ExecutorTypePoolmgr,
|
||||
}
|
||||
|
||||
// create the pool
|
||||
@@ -158,11 +158,6 @@ func MakeGenericPool(
|
||||
|
||||
go gp.choosePodService()
|
||||
|
||||
// Unless specified otherwise, periodically cleanup inactive pods.
|
||||
if env.Spec.AllowedFunctionsPerContainer != fission.AllowedFunctionsPerContainerInfinite {
|
||||
go gp.idlePodReaper()
|
||||
}
|
||||
|
||||
return gp, nil
|
||||
}
|
||||
|
||||
@@ -267,10 +262,10 @@ func (gp *GenericPool) _choosePod(newLabels map[string]string) (*apiv1.Pod, erro
|
||||
|
||||
func (gp *GenericPool) labelsForFunction(metadata *metav1.ObjectMeta) map[string]string {
|
||||
return map[string]string{
|
||||
"functionName": metadata.Name,
|
||||
"functionUid": string(metadata.UID),
|
||||
"unmanaged": "true", // this allows us to easily find pods not managed by the deployment
|
||||
POOLMGR_INSTANCEID_LABEL: gp.instanceId,
|
||||
"functionName": metadata.Name,
|
||||
"functionUid": string(metadata.UID),
|
||||
"unmanaged": "true", // this allows us to easily find pods not managed by the deployment
|
||||
fission.EXECUTOR_INSTANCEID_LABEL: gp.instanceId,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -467,6 +462,7 @@ func (gp *GenericPool) createPool() error {
|
||||
MountPath: gp.sharedMountPath,
|
||||
},
|
||||
},
|
||||
Resources: gp.env.Spec.Resources,
|
||||
},
|
||||
{
|
||||
Name: "fetcher",
|
||||
@@ -582,7 +578,7 @@ func (gp *GenericPool) GetFuncSvc(m *metav1.ObjectMeta) (*fscache.FuncSvc, error
|
||||
|
||||
kubeObjRefs := []api.ObjectReference{
|
||||
{
|
||||
Kind: pod.TypeMeta.Kind,
|
||||
Kind: "pod",
|
||||
Name: pod.ObjectMeta.Name,
|
||||
APIVersion: pod.TypeMeta.APIVersion,
|
||||
Namespace: pod.ObjectMeta.Namespace,
|
||||
@@ -592,83 +588,23 @@ func (gp *GenericPool) GetFuncSvc(m *metav1.ObjectMeta) (*fscache.FuncSvc, error
|
||||
}
|
||||
|
||||
fsvc := &fscache.FuncSvc{
|
||||
Name: pod.ObjectMeta.Name,
|
||||
Function: m,
|
||||
Environment: gp.env,
|
||||
Address: svcHost,
|
||||
KubernetesObjects: kubeObjRefs,
|
||||
Backend: fscache.POOLMGR,
|
||||
Executor: fscache.POOLMGR,
|
||||
Ctime: time.Now(),
|
||||
Atime: time.Now(),
|
||||
}
|
||||
|
||||
err, _ = gp.fsCache.Add(*fsvc)
|
||||
_, err = gp.fsCache.Add(*fsvc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return fsvc, nil
|
||||
}
|
||||
|
||||
func (gp *GenericPool) CleanupFunctionService(obj *fscache.FuncSvc) error {
|
||||
// remove ourselves from fsCache (only if we're still old)
|
||||
deleted, err := gp.fsCache.DeleteOld(obj, gp.idlePodReapTime)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if !deleted {
|
||||
log.Printf("Not deleting %v, in use", obj.Function)
|
||||
return nil
|
||||
}
|
||||
|
||||
for _, kubeobj := range obj.KubernetesObjects {
|
||||
pod, err := gp.kubernetesClient.CoreV1().Pods(gp.namespace).Get(kubeobj.Name, metav1.GetOptions{})
|
||||
|
||||
loggerUrl := fmt.Sprintf("http://%s:1234/v1/log/%s", pod.Spec.NodeName, pod.Name)
|
||||
req, err := http.NewRequest("DELETE", loggerUrl, nil)
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
log.Printf("Error from %s daemonset logger: %v", pod.Spec.NodeName, err)
|
||||
} else {
|
||||
if resp.StatusCode != 200 {
|
||||
log.Printf("Received not http 200(OK) status from %s daemonset logger: %s", pod.Spec.NodeName, resp.Status)
|
||||
}
|
||||
resp.Body.Close()
|
||||
}
|
||||
|
||||
// delete pod
|
||||
err = gp.kubernetesClient.CoreV1().Pods(gp.namespace).Delete(kubeobj.Name, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (gp *GenericPool) idlePodReaper() {
|
||||
for {
|
||||
time.Sleep(time.Minute)
|
||||
funcSvcs, err := gp.fsCache.ListOld(&gp.env.Metadata, gp.idlePodReapTime)
|
||||
if err != nil {
|
||||
log.Printf("Error reaping idle pods: %v", err)
|
||||
continue
|
||||
}
|
||||
for _, obj := range funcSvcs {
|
||||
err := gp.CleanupFunctionService(obj)
|
||||
if err != nil {
|
||||
log.Printf("Error deleting Kubernetes objects for fsvc '%v': %v", obj, err)
|
||||
log.Printf("Object Name| Object Kind | Object Space")
|
||||
for _, kubeobj := range obj.KubernetesObjects {
|
||||
log.Printf("%v | %v | %v", kubeobj.Name, kubeobj.Kind, kubeobj.Namespace)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// destroys the pool -- the deployment, replicaset and pods
|
||||
func (gp *GenericPool) destroy() error {
|
||||
// Destroy deployment
|
||||
|
||||
Reference in New Issue
Block a user