Newdeploy backend (#387)

A newdeploy backend which uses new deployment to serve requests. This is the second phase of #193 and builds on top of changes in #384 .

* Executor layer added on top of pool manager

* Removed the external server for executor

* Minor changes to keep existing semantics as much possible

* Separating the executor vs. poolmgr backend functionality and associated data members

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed references to poolmgr in tests

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Rebased from master and changed references to tpr -> crd

* Executor layer added on top of pool manager

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed podName to a generic objectReference in fscache (#391)

Changed podName to a generic objectReference in function service cache implementation.

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Rebased from master and changed references to tpr -> crd

* Merged from master with latest changes

* Executor layer added on top of pool manager

* Removed the external server for executor

* Minor changes to keep existing semantics as much possible

* Separating the executor vs. poolmgr backend functionality and associated data members

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed references to poolmgr in tests

* update compiling.md to use helm

* Compile instructions: changed pullPolicy to IfNotPresent (#378)

Containers will get stuck in ErrImagePull/ImagePullBackOff state otherwise

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Fetcher called when pod is created for newDeploy backend but also supports older way, this is WIP and still needs pod specialization and creating & exposing a service so the URL can be hit by end user

* WIP Specializing the POD as part of startup along with fetching

* Working specialization of a new deployment. Needs some work on caching, cleanup etc.

* Switched to service based address instead of POD address

* Minor formating issue fixed

* Added logging to pods and a readiness check, the readiness check is flaky though ATM

* Fixed some rebase issues that were failing build

* Better names for K8S objects and methods

* Switched usage of FuncSvc in backends from pod to api.ObjectReference

* Adding retry to fetcher request, for now just using default retry client which might need tweaking in future

* Switching to plain old retry, some issue in getting retryablehttp with glide import

* Removed stale executor service & deployment from previous merge

* Addressed review comments, still testing some areas

* Added types in FunctionSpec

* Resolved conflicts due to merge from executor_abstraction branch

* Added backend type on EnvironmentSpec along with operations for create/list/update, the pools are created/destroyed based on change in backend type

* Backend from types and a minor err return issue fixed

* Draft version of CPU and memory parameters added to environment

* Added resourceReq to newDeploy, though it has some issues

* Issue with resourceName fixed, now newdeploy pods also pick up resources from the environment config

* Adding scale params, removing validation on CPU params for now

* Fixed a formatting issue

* Checking if slight more delay helps in the test which is currently failing for internal routes

* The resourceList newly added in Env can not be compared by compiler, hence must use breakdown comparison instead

* Added strategy selection on client side

* Added caching, informers, delete operations for newdeploy backend functions

* Deleted a stale directory

* A simple HPA based on scale parameters, testing still WIP

* Fixed a small issue in delete function, added HPA delete too when deleting a function

* Previous merge missed the pkg flag for update fn command somehow, fixed that

* Fixed comments from review

* Changed poolmgr cleanup to be generic cleanup and moved to executor, added instanceID labels to newdeploy so that cleanup works

* Moved instanceIdLabel to types to avoid cyclic dependency

* More review fixes

* Tweaking sleep to see results

* If user does not provide poolsize, then it should not default to zero

* Switched to naming convention for now, fixed default poolsize if not provided

* Changed error return behaviour in delete fn, also changed cleanup to look based on obj type though support for additional type will need more work

* Changed check location so avoid false logging

* Test for newdeploy backend

* Adding tests for poolmgr backend

* Fixed an issue with glide dependency version, already fixed in master

* Added instanceId for NewDeploy, Initial cleanup now cleans older objects of newdeploy backend, removed eagercreate flag and instead using minScale to drive eager creation

* Moved cleanup to executor layer with cleanup for newDeploy backend, changes to use the new Cache impl

* Cleaning up pod & rs along with deployment for newdeploy backend

* Enhanced fn and env listing to show min/maxscale and resuorces respectively

* Added conditional heapster deployment and fixed a small issue with resources for fetcher container in function pod

* Addressed review comments from previous change

* Addressed some more review comments - majorly create only on NotFoundError

* Added TargetCPU as an input for scaling

* Bumped target CPU to be greater than 0 and added a default value

* Min replicas should be 1 even if the minScale is 0 when creating deployment

* Changed name from 'backend' to executorType, added additional test for minscale 0 case, changed TargetCPU to TargetCPUPercent

executor/poolmgr/cleanup.go

@@ -1,147 +0,0 @@
-/*
-Copyright 2016 The Fission Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package poolmgr
-
-import (
-	"log"
-	"time"
-
-	meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/client-go/kubernetes"
-)
-
-// cleanupOldPoolmgrResources looks for resources created by an old
-// poolmgr instance and cleans them up.
-func CleanupOldPoolmgrResources(client *kubernetes.Clientset, namespace string, instanceId string) {
-	go func() {
-		err := cleanup(client, namespace, instanceId)
-		if err != nil {
-			// TODO retry cleanup; logged and ignored for now
-			log.Printf("Failed to cleanup: %v", err)
-		}
-	}()
-}
-
-func cleanup(client *kubernetes.Clientset, namespace string, instanceId string) error {
-	// Deployments are used for idle pools and can be cleaned up
-	// immediately.  (We should "adopt" these instead of creating
-	// a new pool.)
-	err := cleanupDeployments(client, namespace, instanceId)
-	if err != nil {
-		return err
-	}
-	// See K8s #33845 and related bugs: deleting a deployment
-	// through the API doesn't cause the associated ReplicaSet to
-	// be deleted.  (Fixed recently, but we may be running a
-	// version before the fix.)
-	err = cleanupReplicaSets(client, namespace, instanceId)
-	if err != nil {
-		return err
-	}
-
-	// Pods might still be running user functions, so we give them
-	// a few minutes before terminating them.  This time is the
-	// maximum function runtime, plus the time a router might
-	// still route to an old instance, i.e. router cache expiry
-	// time.
-	time.Sleep(6 * time.Minute)
-
-	err = cleanupPods(client, namespace, instanceId)
-	if err != nil {
-		return err
-	}
-
-	err = cleanupServices(client, namespace, instanceId)
-	if err != nil {
-		return err
-	}
-
-	return nil
-}
-
-func cleanupDeployments(client *kubernetes.Clientset, namespace string, instanceId string) error {
-	deploymentList, err := client.ExtensionsV1beta1().Deployments(namespace).List(meta_v1.ListOptions{})
-	if err != nil {
-		return err
-	}
-	for _, dep := range deploymentList.Items {
-		id, ok := dep.ObjectMeta.Labels[POOLMGR_INSTANCEID_LABEL]
-		if ok && id != instanceId {
-			log.Printf("Cleaning up deployment %v", dep.ObjectMeta.Name)
-			err := client.ExtensionsV1beta1().Deployments(namespace).Delete(dep.ObjectMeta.Name, nil)
-			logErr("cleaning up deployment", err)
-			// ignore err
-		}
-	}
-	return nil
-}
-
-func cleanupReplicaSets(client *kubernetes.Clientset, namespace string, instanceId string) error {
-	rsList, err := client.ExtensionsV1beta1().ReplicaSets(namespace).List(meta_v1.ListOptions{})
-	if err != nil {
-		return err
-	}
-	for _, rs := range rsList.Items {
-		id, ok := rs.ObjectMeta.Labels[POOLMGR_INSTANCEID_LABEL]
-		if ok && id != instanceId {
-			log.Printf("Cleaning up replicaset %v", rs.ObjectMeta.Name)
-			err := client.ExtensionsV1beta1().ReplicaSets(namespace).Delete(rs.ObjectMeta.Name, nil)
-			logErr("cleaning up replicaset", err)
-		}
-	}
-	return nil
-}
-
-func cleanupPods(client *kubernetes.Clientset, namespace string, instanceId string) error {
-	podList, err := client.CoreV1().Pods(namespace).List(meta_v1.ListOptions{})
-	if err != nil {
-		return err
-	}
-	for _, pod := range podList.Items {
-		id, ok := pod.ObjectMeta.Labels[POOLMGR_INSTANCEID_LABEL]
-		if ok && id != instanceId {
-			log.Printf("Cleaning up pod %v", pod.ObjectMeta.Name)
-			err := client.CoreV1().Pods(namespace).Delete(pod.ObjectMeta.Name, nil)
-			logErr("cleaning up pod", err)
-			// ignore err
-		}
-	}
-	return nil
-}
-
-func cleanupServices(client *kubernetes.Clientset, namespace string, instanceId string) error {
-	svcList, err := client.CoreV1().Services(namespace).List(meta_v1.ListOptions{})
-	if err != nil {
-		return err
-	}
-	for _, svc := range svcList.Items {
-		id, ok := svc.ObjectMeta.Labels[POOLMGR_INSTANCEID_LABEL]
-		if ok && id != instanceId {
-			log.Printf("Cleaning up svc %v", svc.ObjectMeta.Name)
-			err := client.CoreV1().Services(namespace).Delete(svc.ObjectMeta.Name, nil)
-			logErr("cleaning up svc", err)
-			// ignore err
-		}
-	}
-	return nil
-}
-
-func logErr(msg string, err error) {
-	if err != nil {
-		log.Printf("Error %v: %v", msg, err)
-	}
-}

executor/poolmgr/gp.go

@@ -47,7 +47,6 @@ import (
 	"github.com/fission/fission/executor/fscache"
 )
 
-const POOLMGR_INSTANCEID_LABEL string = "poolmgrInstanceId"
 const POD_PHASE_RUNNING string = "Running"
 
 type (
@@ -144,9 +143,10 @@ func MakeGenericPool(
 
 	// Labels for generic deployment/RS/pods.
 	gp.labelsForPool = map[string]string{
-		"environmentName":        gp.env.Metadata.Name,
-		"environmentUid":         string(gp.env.Metadata.UID),
-		POOLMGR_INSTANCEID_LABEL: gp.instanceId,
+		"environmentName":                 gp.env.Metadata.Name,
+		"environmentUid":                  string(gp.env.Metadata.UID),
+		fission.EXECUTOR_INSTANCEID_LABEL: gp.instanceId,
+		"executorType":                    fission.ExecutorTypePoolmgr,
 	}
 
 	// create the pool
@@ -158,11 +158,6 @@ func MakeGenericPool(
 
 	go gp.choosePodService()
 
-	// Unless specified otherwise, periodically cleanup inactive pods.
-	if env.Spec.AllowedFunctionsPerContainer != fission.AllowedFunctionsPerContainerInfinite {
-		go gp.idlePodReaper()
-	}
-
 	return gp, nil
 }
 
@@ -267,10 +262,10 @@ func (gp *GenericPool) _choosePod(newLabels map[string]string) (*apiv1.Pod, erro
 
 func (gp *GenericPool) labelsForFunction(metadata *metav1.ObjectMeta) map[string]string {
 	return map[string]string{
-		"functionName":           metadata.Name,
-		"functionUid":            string(metadata.UID),
-		"unmanaged":              "true", // this allows us to easily find pods not managed by the deployment
-		POOLMGR_INSTANCEID_LABEL: gp.instanceId,
+		"functionName":                    metadata.Name,
+		"functionUid":                     string(metadata.UID),
+		"unmanaged":                       "true", // this allows us to easily find pods not managed by the deployment
+		fission.EXECUTOR_INSTANCEID_LABEL: gp.instanceId,
 	}
 }
 
@@ -467,6 +462,7 @@ func (gp *GenericPool) createPool() error {
 									MountPath: gp.sharedMountPath,
 								},
 							},
+							Resources: gp.env.Spec.Resources,
 						},
 						{
 							Name:                   "fetcher",
@@ -582,7 +578,7 @@ func (gp *GenericPool) GetFuncSvc(m *metav1.ObjectMeta) (*fscache.FuncSvc, error
 
 	kubeObjRefs := []api.ObjectReference{
 		{
-			Kind:            pod.TypeMeta.Kind,
+			Kind:            "pod",
 			Name:            pod.ObjectMeta.Name,
 			APIVersion:      pod.TypeMeta.APIVersion,
 			Namespace:       pod.ObjectMeta.Namespace,
@@ -592,83 +588,23 @@ func (gp *GenericPool) GetFuncSvc(m *metav1.ObjectMeta) (*fscache.FuncSvc, error
 	}
 
 	fsvc := &fscache.FuncSvc{
+		Name:              pod.ObjectMeta.Name,
 		Function:          m,
 		Environment:       gp.env,
 		Address:           svcHost,
 		KubernetesObjects: kubeObjRefs,
-		Backend:           fscache.POOLMGR,
+		Executor:          fscache.POOLMGR,
 		Ctime:             time.Now(),
 		Atime:             time.Now(),
 	}
 
-	err, _ = gp.fsCache.Add(*fsvc)
+	_, err = gp.fsCache.Add(*fsvc)
 	if err != nil {
 		return nil, err
 	}
 	return fsvc, nil
 }
 
-func (gp *GenericPool) CleanupFunctionService(obj *fscache.FuncSvc) error {
-	// remove ourselves from fsCache (only if we're still old)
-	deleted, err := gp.fsCache.DeleteOld(obj, gp.idlePodReapTime)
-	if err != nil {
-		return err
-	}
-
-	if !deleted {
-		log.Printf("Not deleting %v, in use", obj.Function)
-		return nil
-	}
-
-	for _, kubeobj := range obj.KubernetesObjects {
-		pod, err := gp.kubernetesClient.CoreV1().Pods(gp.namespace).Get(kubeobj.Name, metav1.GetOptions{})
-
-		loggerUrl := fmt.Sprintf("http://%s:1234/v1/log/%s", pod.Spec.NodeName, pod.Name)
-		req, err := http.NewRequest("DELETE", loggerUrl, nil)
-		resp, err := http.DefaultClient.Do(req)
-		if err != nil {
-			log.Printf("Error from %s daemonset logger: %v", pod.Spec.NodeName, err)
-		} else {
-			if resp.StatusCode != 200 {
-				log.Printf("Received not http 200(OK) status from %s daemonset logger: %s", pod.Spec.NodeName, resp.Status)
-			}
-			resp.Body.Close()
-		}
-
-		// delete pod
-		err = gp.kubernetesClient.CoreV1().Pods(gp.namespace).Delete(kubeobj.Name, nil)
-		if err != nil {
-			return err
-		}
-		if err != nil {
-			return err
-		}
-	}
-
-	return nil
-}
-
-func (gp *GenericPool) idlePodReaper() {
-	for {
-		time.Sleep(time.Minute)
-		funcSvcs, err := gp.fsCache.ListOld(&gp.env.Metadata, gp.idlePodReapTime)
-		if err != nil {
-			log.Printf("Error reaping idle pods: %v", err)
-			continue
-		}
-		for _, obj := range funcSvcs {
-			err := gp.CleanupFunctionService(obj)
-			if err != nil {
-				log.Printf("Error deleting Kubernetes objects for fsvc '%v': %v", obj, err)
-				log.Printf("Object Name| Object Kind | Object Space")
-				for _, kubeobj := range obj.KubernetesObjects {
-					log.Printf("%v | %v | %v", kubeobj.Name, kubeobj.Kind, kubeobj.Namespace)
-				}
-			}
-		}
-	}
-}
-
 // destroys the pool -- the deployment, replicaset and pods
 func (gp *GenericPool) destroy() error {
 	// Destroy deployment

executor/poolmgr/gpm.go

@@ -89,7 +89,7 @@ func (gpm *GenericPoolManager) service() {
 			var err error
 			pool, ok := gpm.pools[crd.CacheKey(&req.env.Metadata)]
 			if !ok {
-				var poolSize int32 = 3 // TODO configurable/autoscalable
+				var poolSize = int32(req.env.Spec.Poolsize)
 				switch req.env.Spec.AllowedFunctionsPerContainer {
 				case fission.AllowedFunctionsPerContainerInfinite:
 					poolSize = 1
@@ -107,13 +107,17 @@ func (gpm *GenericPoolManager) service() {
 			req.responseChannel <- &response{pool: pool}
 		case CLEANUP_POOLS:
 			latestEnvSet := make(map[string]bool)
+			latestEnvPoolsize := make(map[string]int)
 			for _, env := range req.envList {
 				latestEnvSet[crd.CacheKey(&env.Metadata)] = true
+				latestEnvPoolsize[crd.CacheKey(&env.Metadata)] = env.Spec.Poolsize
 			}
 			for key, pool := range gpm.pools {
 				_, ok := latestEnvSet[key]
-				if !ok {
-					// Env no longer exists -- remove our cache
+				poolsize := latestEnvPoolsize[key]
+				if !ok || poolsize == 0 {
+					// Env no longer exists or pool size changed to zero
+
 					log.Printf("Destroying generic pool for environment [%v]", key)
 					delete(gpm.pools, key)
 
@@ -160,9 +164,13 @@ func (gpm *GenericPoolManager) eagerPoolCreator() {
 		// to keep these eagerly created pools smaller than the ones created when there are
 		// actual function calls.
 		for i := range envs.Items {
-			_, err := gpm.GetPool(&envs.Items[i])
-			if err != nil {
-				log.Printf("eager-create pool failed: %v", err)
+			env := envs.Items[i]
+			// Create pool only if poolsize greater than zero
+			if env.Spec.Poolsize > 0 {
+				_, err := gpm.GetPool(&envs.Items[i])
+				if err != nil {
+					log.Printf("eager-create pool failed: %v", err)
+				}
 			}
 		}