Newdeploy backend (#387)

A newdeploy backend which uses new deployment to serve requests. This is the second phase of #193 and builds on top of changes in #384 .

* Executor layer added on top of pool manager

* Removed the external server for executor

* Minor changes to keep existing semantics as much possible

* Separating the executor vs. poolmgr backend functionality and associated data members

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed references to poolmgr in tests

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Rebased from master and changed references to tpr -> crd

* Executor layer added on top of pool manager

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed podName to a generic objectReference in fscache (#391)

Changed podName to a generic objectReference in function service cache implementation.

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Rebased from master and changed references to tpr -> crd

* Merged from master with latest changes

* Executor layer added on top of pool manager

* Removed the external server for executor

* Minor changes to keep existing semantics as much possible

* Separating the executor vs. poolmgr backend functionality and associated data members

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed references to poolmgr in tests

* update compiling.md to use helm

* Compile instructions: changed pullPolicy to IfNotPresent (#378)

Containers will get stuck in ErrImagePull/ImagePullBackOff state otherwise

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Fetcher called when pod is created for newDeploy backend but also supports older way, this is WIP and still needs pod specialization and creating & exposing a service so the URL can be hit by end user

* WIP Specializing the POD as part of startup along with fetching

* Working specialization of a new deployment. Needs some work on caching, cleanup etc.

* Switched to service based address instead of POD address

* Minor formating issue fixed

* Added logging to pods and a readiness check, the readiness check is flaky though ATM

* Fixed some rebase issues that were failing build

* Better names for K8S objects and methods

* Switched usage of FuncSvc in backends from pod to api.ObjectReference

* Adding retry to fetcher request, for now just using default retry client which might need tweaking in future

* Switching to plain old retry, some issue in getting retryablehttp with glide import

* Removed stale executor service & deployment from previous merge

* Addressed review comments, still testing some areas

* Added types in FunctionSpec

* Resolved conflicts due to merge from executor_abstraction branch

* Added backend type on EnvironmentSpec along with operations for create/list/update, the pools are created/destroyed based on change in backend type

* Backend from types and a minor err return issue fixed

* Draft version of CPU and memory parameters added to environment

* Added resourceReq to newDeploy, though it has some issues

* Issue with resourceName fixed, now newdeploy pods also pick up resources from the environment config

* Adding scale params, removing validation on CPU params for now

* Fixed a formatting issue

* Checking if slight more delay helps in the test which is currently failing for internal routes

* The resourceList newly added in Env can not be compared by compiler, hence must use breakdown comparison instead

* Added strategy selection on client side

* Added caching, informers, delete operations for newdeploy backend functions

* Deleted a stale directory

* A simple HPA based on scale parameters, testing still WIP

* Fixed a small issue in delete function, added HPA delete too when deleting a function

* Previous merge missed the pkg flag for update fn command somehow, fixed that

* Fixed comments from review

* Changed poolmgr cleanup to be generic cleanup and moved to executor, added instanceID labels to newdeploy so that cleanup works

* Moved instanceIdLabel to types to avoid cyclic dependency

* More review fixes

* Tweaking sleep to see results

* If user does not provide poolsize, then it should not default to zero

* Switched to naming convention for now, fixed default poolsize if not provided

* Changed error return behaviour in delete fn, also changed cleanup to look based on obj type though support for additional type will need more work

* Changed check location so avoid false logging

* Test for newdeploy backend

* Adding tests for poolmgr backend

* Fixed an issue with glide dependency version, already fixed in master

* Added instanceId for NewDeploy, Initial cleanup now cleans older objects of newdeploy backend, removed eagercreate flag and instead using minScale to drive eager creation

* Moved cleanup to executor layer with cleanup for newDeploy backend, changes to use the new Cache impl

* Cleaning up pod & rs along with deployment for newdeploy backend

* Enhanced fn and env listing to show min/maxscale and resuorces respectively

* Added conditional heapster deployment and fixed a small issue with resources for fetcher container in function pod

* Addressed review comments from previous change

* Addressed some more review comments - majorly create only on NotFoundError

* Added TargetCPU as an input for scaling

* Bumped target CPU to be greater than 0 and added a default value

* Min replicas should be 1 even if the minScale is 0 when creating deployment

* Changed name from 'backend' to executorType, added additional test for minscale 0 case, changed TargetCPU to TargetCPUPercent

executor/api.go

@@ -17,6 +17,7 @@ limitations under the License.
 package executor
 
 import (
+	"context"
 	"encoding/json"
 	"fmt"
 	"io/ioutil"
@@ -104,5 +105,8 @@ func (executor *Executor) Serve(port int) {
 	r.HandleFunc("/v2/tapService", executor.tapService).Methods("POST")
 	address := fmt.Sprintf(":%v", port)
 	log.Printf("starting executor at port %v", port)
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	executor.ndm.Run(ctx)
 	log.Fatal(http.ListenAndServe(address, handlers.LoggingHandler(os.Stdout, r)))
 }

executor/cleanup.go

@@ -0,0 +1,282 @@
+/*
+Copyright 2016 The Fission Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package executor
+
+import (
+	"fmt"
+	"log"
+	"strings"
+	"time"
+
+	meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/pkg/api"
+
+	"github.com/fission/fission"
+	"github.com/fission/fission/crd"
+	"github.com/fission/fission/executor/fscache"
+)
+
+// cleanupObjects cleans up resources created by old executortype instances
+func cleanupObjects(kubernetesClient *kubernetes.Clientset,
+	namespace string,
+	instanceId string) {
+	go func() {
+		err := cleanup(kubernetesClient, namespace, instanceId)
+		if err != nil {
+			// TODO retry cleanup; logged and ignored for now
+			log.Printf("Failed to cleanup: %v", err)
+		}
+	}()
+}
+
+func cleanup(client *kubernetes.Clientset, namespace string, instanceId string) error {
+
+	err := cleanupServices(client, namespace, instanceId)
+	if err != nil {
+		return err
+	}
+
+	err = cleanupHpa(client, namespace, instanceId)
+	if err != nil {
+		return err
+	}
+
+	// Deployments are used for idle pools and can be cleaned up
+	// immediately.  (We should "adopt" these instead of creating
+	// a new pool.)
+	err = cleanupDeployments(client, namespace, instanceId)
+	if err != nil {
+		return err
+	}
+	// See K8s #33845 and related bugs: deleting a deployment
+	// through the API doesn't cause the associated ReplicaSet to
+	// be deleted.  (Fixed recently, but we may be running a
+	// version before the fix.)
+	err = cleanupReplicaSets(client, namespace, instanceId)
+	if err != nil {
+		return err
+	}
+
+	// Pods might still be running user functions, so we give them
+	// a few minutes before terminating them.  This time is the
+	// maximum function runtime, plus the time a router might
+	// still route to an old instance, i.e. router cache expiry
+	// time.
+	time.Sleep(6 * time.Minute)
+
+	err = cleanupPods(client, namespace, instanceId)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// idleObjectReaper reaps objects after certain idle time
+func idleObjectReaper(kubeClient *kubernetes.Clientset,
+	fissionClient *crd.FissionClient,
+	fsCache *fscache.FunctionServiceCache,
+	idlePodReapTime time.Duration) {
+
+	pollSleep := time.Duration(2 * time.Minute)
+	for {
+		time.Sleep(pollSleep)
+
+		envs, err := fissionClient.Environments(meta_v1.NamespaceAll).List(meta_v1.ListOptions{})
+		if err != nil {
+			log.Fatalf("Failed to get environment list: %v", err)
+		}
+
+		for i := range envs.Items {
+			env := envs.Items[i]
+			if env.Spec.AllowedFunctionsPerContainer == fission.AllowedFunctionsPerContainerInfinite {
+				continue
+			}
+			funcSvcs, err := fsCache.ListOld(&env.Metadata, idlePodReapTime)
+			if err != nil {
+				log.Printf("Error reaping idle pods: %v", err)
+				continue
+			}
+
+			for _, fsvc := range funcSvcs {
+
+				fn, err := fissionClient.Functions(fsvc.Function.Namespace).Get(fsvc.Function.Name)
+				if err != nil {
+					log.Printf("Error getting function: %v", fsvc.Function.Name)
+					continue
+				}
+
+				// Ignore functions of NewDeploy ExecutorType with MinScale > 0
+				if fn.Spec.InvokeStrategy.ExecutionStrategy.MinScale > 0 && fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType == fission.ExecutorTypeNewdeploy {
+					continue
+				}
+				deleted, err := fsCache.DeleteOld(fsvc, idlePodReapTime)
+
+				if err != nil {
+					log.Printf("Error deleting Kubernetes objects for fsvc '%v': %v", fsvc, err)
+					log.Printf("Object Name| Object Kind | Object Namespace")
+					for _, kubeobj := range fsvc.KubernetesObjects {
+						log.Printf("%v | %v | %v", kubeobj.Name, kubeobj.Kind, kubeobj.Namespace)
+					}
+				}
+
+				if !deleted {
+					continue
+				}
+				for _, kubeobj := range fsvc.KubernetesObjects {
+					deleteKubeobject(kubeClient, &kubeobj)
+				}
+			}
+		}
+	}
+}
+
+func deleteKubeobject(kubeClient *kubernetes.Clientset, kubeobj *api.ObjectReference) {
+	switch strings.ToLower(kubeobj.Kind) {
+	case "pod":
+		err := kubeClient.CoreV1().Pods(kubeobj.Namespace).Delete(kubeobj.Name, nil)
+		logErr(fmt.Sprintf("cleaning up pod %v ", kubeobj.Name), err)
+
+	case "service":
+		err := kubeClient.CoreV1().Services(kubeobj.Namespace).Delete(kubeobj.Name, nil)
+		logErr(fmt.Sprintf("cleaning up service %v ", kubeobj.Name), err)
+
+	case "deployment":
+		depl, err := kubeClient.ExtensionsV1beta1().Deployments(kubeobj.Namespace).Get(kubeobj.Name, meta_v1.GetOptions{})
+		err = kubeClient.ExtensionsV1beta1().Deployments(kubeobj.Namespace).Delete(kubeobj.Name, nil)
+		logErr(fmt.Sprintf("cleaning up deployment %v ", kubeobj.Name), err)
+		cleanupDeploymentObjects(kubeClient, kubeobj.Namespace, depl.Labels)
+
+	case "horizontalpodautoscaler":
+		err := kubeClient.AutoscalingV1().HorizontalPodAutoscalers(kubeobj.Namespace).Delete(kubeobj.Name, nil)
+		logErr(fmt.Sprintf("cleaning up horizontalpodautoscaler %v ", kubeobj.Name), err)
+
+	default:
+		log.Printf("There was an error identifying the object type: %v for obj: %v", kubeobj.Kind, kubeobj)
+
+	}
+}
+
+func cleanupDeploymentObjects(kubeClient *kubernetes.Clientset, namespace string, sel map[string]string) {
+	rsList, err := kubeClient.ExtensionsV1beta1().ReplicaSets(namespace).List(meta_v1.ListOptions{LabelSelector: labels.Set(sel).AsSelector().String()})
+	logErr("Getting replicaset for deployment ", err)
+	for _, rs := range rsList.Items {
+		err = kubeClient.ExtensionsV1beta1().ReplicaSets(namespace).Delete(rs.Name, nil)
+		logErr(fmt.Sprintf("Cleaning replicaset %v for deployment", rs.Name), err)
+	}
+
+	podList, err := kubeClient.CoreV1().Pods(namespace).List(meta_v1.ListOptions{LabelSelector: labels.Set(sel).AsSelector().String()})
+	logErr("Getting pods for deployment ", err)
+	for _, pod := range podList.Items {
+		err = kubeClient.CoreV1().Pods(namespace).Delete(pod.Name, nil)
+		logErr(fmt.Sprintf("Cleaning pod %v for deployment", pod.Name), err)
+	}
+}
+
+func cleanupDeployments(client *kubernetes.Clientset, namespace string, instanceId string) error {
+	deploymentList, err := client.ExtensionsV1beta1().Deployments(namespace).List(meta_v1.ListOptions{})
+	if err != nil {
+		return err
+	}
+	for _, dep := range deploymentList.Items {
+		id, ok := dep.ObjectMeta.Labels[fission.EXECUTOR_INSTANCEID_LABEL]
+		if ok && id != instanceId {
+			log.Printf("Cleaning up deployment %v", dep.ObjectMeta.Name)
+			err := client.ExtensionsV1beta1().Deployments(namespace).Delete(dep.ObjectMeta.Name, nil)
+			logErr("cleaning up deployment", err)
+			// ignore err
+		}
+	}
+	return nil
+}
+
+func cleanupReplicaSets(client *kubernetes.Clientset, namespace string, instanceId string) error {
+	rsList, err := client.ExtensionsV1beta1().ReplicaSets(namespace).List(meta_v1.ListOptions{})
+	if err != nil {
+		return err
+	}
+	for _, rs := range rsList.Items {
+		id, ok := rs.ObjectMeta.Labels[fission.EXECUTOR_INSTANCEID_LABEL]
+		if ok && id != instanceId {
+			log.Printf("Cleaning up replicaset %v", rs.ObjectMeta.Name)
+			err := client.ExtensionsV1beta1().ReplicaSets(namespace).Delete(rs.ObjectMeta.Name, nil)
+			logErr("cleaning up replicaset", err)
+		}
+	}
+	return nil
+}
+
+func cleanupPods(client *kubernetes.Clientset, namespace string, instanceId string) error {
+	podList, err := client.CoreV1().Pods(namespace).List(meta_v1.ListOptions{})
+	if err != nil {
+		return err
+	}
+	for _, pod := range podList.Items {
+		id, ok := pod.ObjectMeta.Labels[fission.EXECUTOR_INSTANCEID_LABEL]
+		if ok && id != instanceId {
+			log.Printf("Cleaning up pod %v", pod.ObjectMeta.Name)
+			err := client.CoreV1().Pods(namespace).Delete(pod.ObjectMeta.Name, nil)
+			logErr("cleaning up pod", err)
+			// ignore err
+		}
+	}
+	return nil
+}
+
+func cleanupServices(client *kubernetes.Clientset, namespace string, instanceId string) error {
+	svcList, err := client.CoreV1().Services(namespace).List(meta_v1.ListOptions{})
+	if err != nil {
+		return err
+	}
+	for _, svc := range svcList.Items {
+		id, ok := svc.ObjectMeta.Labels[fission.EXECUTOR_INSTANCEID_LABEL]
+		if ok && id != instanceId {
+			log.Printf("Cleaning up svc %v", svc.ObjectMeta.Name)
+			err := client.CoreV1().Services(namespace).Delete(svc.ObjectMeta.Name, nil)
+			logErr("cleaning up service", err)
+			// ignore err
+		}
+	}
+	return nil
+}
+
+func cleanupHpa(client *kubernetes.Clientset, namespace string, instanceId string) error {
+	hpaList, err := client.AutoscalingV1().HorizontalPodAutoscalers(namespace).List(meta_v1.ListOptions{})
+	if err != nil {
+		return err
+	}
+
+	for _, hpa := range hpaList.Items {
+		id, ok := hpa.ObjectMeta.Labels[fission.EXECUTOR_INSTANCEID_LABEL]
+		if ok && id != instanceId {
+			log.Printf("Cleaning up HPA %v", hpa.ObjectMeta.Name)
+			err := client.AutoscalingV1().HorizontalPodAutoscalers(namespace).Delete(hpa.ObjectMeta.Name, nil)
+			logErr("cleaning up HPA", err)
+		}
+
+	}
+	return nil
+
+}
+
+func logErr(msg string, err error) {
+	if err != nil {
+		log.Printf("Error %v: %v", msg, err)
+	}
+}

executor/deploymgr/newdeploy.go

@@ -1,22 +0,0 @@
-/*
-Copyright 2016 The Fission Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package deploymgr
-
-func GetFuncSvc() (*funcSvc, error) {
-
-	return nil, nil
-}

executor/executor.go

@@ -18,22 +18,25 @@ package executor
 
 import (
 	"log"
-	"os"
+	"strings"
 	"sync"
 	"time"
 
 	"github.com/dchest/uniuri"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
+	"github.com/fission/fission"
 	"github.com/fission/fission/cache"
 	"github.com/fission/fission/crd"
 	"github.com/fission/fission/executor/fscache"
+	"github.com/fission/fission/executor/newdeploy"
 	"github.com/fission/fission/executor/poolmgr"
 )
 
 type (
 	Executor struct {
 		gpm           *poolmgr.GenericPoolManager
+		ndm           *newdeploy.NewDeploy
 		functionEnv   *cache.Cache
 		fissionClient *crd.FissionClient
 		fsCache       *fscache.FunctionServiceCache
@@ -52,9 +55,10 @@ type (
 	}
 )
 
-func MakeExecutor(gpm *poolmgr.GenericPoolManager, fissionClient *crd.FissionClient, fsCache *fscache.FunctionServiceCache) *Executor {
+func MakeExecutor(gpm *poolmgr.GenericPoolManager, ndm *newdeploy.NewDeploy, fissionClient *crd.FissionClient, fsCache *fscache.FunctionServiceCache) *Executor {
 	executor := &Executor{
 		gpm:           gpm,
+		ndm:           ndm,
 		functionEnv:   cache.MakeCache(10*time.Second, 0),
 		fissionClient: fissionClient,
 		fsCache:       fsCache,
@@ -114,18 +118,27 @@ func (executor *Executor) serveCreateFuncServices() {
 	}
 }
 
-func (executor *Executor) createServiceForFunction(m *metav1.ObjectMeta) (*fscache.FuncSvc, error) {
-	log.Printf("[%v] No cached function service found, creating one", m.Name)
+func (executor *Executor) createServiceForFunction(meta *metav1.ObjectMeta) (*fscache.FuncSvc, error) {
+	log.Printf("[%v] No cached function service found, creating one", meta.Name)
 
-	env, err := executor.getFunctionEnv(m)
+	// from Func -> get Env
+	log.Printf("[%v] getting environment for function", meta.Name)
+	env, err := executor.getFunctionEnv(meta)
 	if err != nil {
 		return nil, err
 	}
-	// Appropriate backend handles the service creation
-	backend := os.Getenv("EXECUTOR_BACKEND")
-	switch backend {
-	case "DEPLOY":
-		return nil, nil
+
+	fn, err := executor.fissionClient.
+		Functions(meta.Namespace).
+		Get(meta.Name)
+	if err != nil {
+		return nil, err
+	}
+
+	switch fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType {
+	case fission.ExecutorTypeNewdeploy:
+		fs, err := executor.ndm.GetFuncSvc(meta)
+		return fs, err
 	default:
 		pool, err := executor.gpm.GetPool(env)
 		if err != nil {
@@ -133,12 +146,9 @@ func (executor *Executor) createServiceForFunction(m *metav1.ObjectMeta) (*fscac
 		}
 		// from GenericPool -> get one function container
 		// (this also adds to the cache)
-		log.Printf("[%v] getting function service from pool", m.Name)
-		fsvc, err := pool.GetFuncSvc(m)
-		if err != nil {
-			return nil, err
-		}
-		return fsvc, nil
+		log.Printf("[%v] getting function service from pool", meta.Name)
+		fsvc, err := pool.GetFuncSvc(meta)
+		return fsvc, err
 	}
 }
 
@@ -171,24 +181,31 @@ func (executor *Executor) getFunctionEnv(m *metav1.ObjectMeta) (*crd.Environment
 	return env, nil
 }
 
-// StartExecutor Starts executor and the backend components that executor uses such as Poolmgr,
-// deploymgr and potential future backends
+// StartExecutor Starts executor and the executor components such as Poolmgr,
+// deploymgr and potential future executor types
 func StartExecutor(fissionNamespace string, functionNamespace string, port int) error {
 	fissionClient, kubernetesClient, _, err := crd.MakeFissionClient()
+	restClient := fissionClient.GetCrdClient()
 	if err != nil {
 		log.Printf("Failed to get kubernetes client: %v", err)
 		return err
 	}
 
-	instanceID := uniuri.NewLen(8)
-	poolmgr.CleanupOldPoolmgrResources(kubernetesClient, functionNamespace, instanceID)
-
 	fsCache := fscache.MakeFunctionServiceCache()
+
+	poolID := strings.ToLower(uniuri.NewLen(8))
+	cleanupObjects(kubernetesClient, functionNamespace, poolID)
+	go idleObjectReaper(kubernetesClient, fissionClient, fsCache, time.Minute*2)
 	gpm := poolmgr.MakeGenericPoolManager(
 		fissionClient, kubernetesClient, fissionNamespace,
-		functionNamespace, fsCache, instanceID)
+		functionNamespace, fsCache, poolID)
+
+	ndm := newdeploy.MakeNewDeploy(
+		fissionClient, kubernetesClient, restClient,
+		functionNamespace, fsCache, poolID)
+
+	api := MakeExecutor(gpm, ndm, fissionClient, fsCache)
 
-	api := MakeExecutor(gpm, fissionClient, fsCache)
 	go api.Serve(port)
 
 	return nil

executor/fscache/functionServiceCache.go

@@ -29,7 +29,7 @@ import (
 )
 
 type fscRequestType int
-type backendType int
+type executorType int
 
 const (
 	TOUCH fscRequestType = iota
@@ -38,17 +38,18 @@ const (
 )
 
 const (
-	POOLMGR backendType = iota
+	POOLMGR executorType = iota
 	NEWDEPLOY
 )
 
 type (
 	FuncSvc struct {
+		Name              string                // Name of object
 		Function          *metav1.ObjectMeta    // function this pod/service is for
 		Environment       *crd.Environment      // function's environment
 		Address           string                // Host:Port or IP:Port that the function's service can be reached at.
 		KubernetesObjects []api.ObjectReference // Kubernetes Objects (within the function namespace)
-		Backend           backendType
+		Executor          executorType
 
 		Ctime time.Time
 		Atime time.Time
@@ -135,20 +136,19 @@ func (fsc *FunctionServiceCache) GetByFunction(m *metav1.ObjectMeta) (*FuncSvc,
 	return &fsvcCopy, nil
 }
 
-// TODO: error should be second return
-func (fsc *FunctionServiceCache) Add(fsvc FuncSvc) (error, *FuncSvc) {
+func (fsc *FunctionServiceCache) Add(fsvc FuncSvc) (*FuncSvc, error) {
 	err, existing := fsc.byFunction.Set(crd.CacheKey(fsvc.Function), &fsvc)
 	if err != nil {
 		if existing != nil {
 			f := existing.(*FuncSvc)
 			err2 := fsc.TouchByAddress(f.Address)
 			if err2 != nil {
-				return err2, nil
+				return nil, err2
 			}
 			fCopy := *f
-			return err, &fCopy
+			return &fCopy, err
 		}
-		return err, nil
+		return nil, err
 	}
 	now := time.Now()
 	fsvc.Ctime = now
@@ -164,7 +164,7 @@ func (fsc *FunctionServiceCache) Add(fsvc FuncSvc) (error, *FuncSvc) {
 			}
 		}
 		log.Printf("error caching fsvc: %v", err)
-		return err, nil
+		return nil, err
 	}
 	return nil, nil
 }

executor/fscache/functionServiceCache_test.go

@@ -59,7 +59,7 @@ func TestFunctionServiceCache(t *testing.T) {
 		Ctime:             now,
 		Atime:             now,
 	}
-	err, _ := fsc.Add(*fsvc)
+	_, err := fsc.Add(*fsvc)
 	if err != nil {
 		fsc.Log()
 		log.Panicf("Failed to add fsvc: %v", err)

executor/newdeploy/newdeploy.go

@@ -0,0 +1,296 @@
+/*
+Copyright 2016 The Fission Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package newdeploy
+
+import (
+	"encoding/json"
+	"errors"
+	"log"
+	"path/filepath"
+	"strconv"
+	"time"
+
+	k8s_err "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/util/intstr"
+	apiv1 "k8s.io/client-go/pkg/api/v1"
+	asv1 "k8s.io/client-go/pkg/apis/autoscaling/v1"
+	"k8s.io/client-go/pkg/apis/extensions/v1beta1"
+
+	"github.com/fission/fission"
+	"github.com/fission/fission/crd"
+	"github.com/fission/fission/environments/fetcher"
+)
+
+const (
+	DeploymentKind    = "Deployment"
+	DeploymentVersion = "extensions/v1beta1"
+)
+
+const (
+	envVersion = "ENV_VERSION"
+)
+
+func (deploy *NewDeploy) createOrGetDeployment(fn *crd.Function, env *crd.Environment,
+	deployName string, deployLabels map[string]string) (*v1beta1.Deployment, error) {
+
+	replicas := int32(fn.Spec.InvokeStrategy.ExecutionStrategy.MinScale)
+	if replicas == 0 {
+		replicas = 1
+	}
+	targetFilename := "user"
+	userfunc := "userfunc"
+
+	existingDepl, err := deploy.kubernetesClient.ExtensionsV1beta1().Deployments(deploy.namespace).Get(deployName, metav1.GetOptions{})
+	if err == nil && existingDepl.Status.ReadyReplicas >= replicas {
+		return existingDepl, err
+	}
+
+	if err != nil && k8s_err.IsNotFound(err) {
+		fetchReq := &fetcher.FetchRequest{
+			FetchType: fetcher.FETCH_DEPLOYMENT,
+			Package: metav1.ObjectMeta{
+				Namespace: fn.Spec.Package.PackageRef.Namespace,
+				Name:      fn.Spec.Package.PackageRef.Name,
+			},
+			Filename: targetFilename,
+		}
+
+		loadReq := fission.FunctionLoadRequest{
+			FilePath:         filepath.Join(deploy.sharedMountPath, targetFilename),
+			FunctionName:     fn.Spec.Package.FunctionName,
+			FunctionMetadata: &fn.Metadata,
+		}
+
+		fetchPayload, err := json.Marshal(fetchReq)
+		if err != nil {
+			return nil, err
+		}
+		loadPayload, err := json.Marshal(loadReq)
+		if err != nil {
+			return nil, err
+		}
+
+		deployment := &v1beta1.Deployment{
+			ObjectMeta: metav1.ObjectMeta{
+				Labels: deployLabels,
+				Name:   deployName,
+			},
+			Spec: v1beta1.DeploymentSpec{
+				Replicas: &replicas,
+				Selector: &metav1.LabelSelector{
+					MatchLabels: deployLabels,
+				},
+				Template: apiv1.PodTemplateSpec{
+					ObjectMeta: metav1.ObjectMeta{
+						Labels: deployLabels,
+					},
+					Spec: apiv1.PodSpec{
+						Volumes: []apiv1.Volume{
+							{
+								Name: userfunc,
+								VolumeSource: apiv1.VolumeSource{
+									EmptyDir: &apiv1.EmptyDirVolumeSource{},
+								},
+							},
+						},
+						Containers: []apiv1.Container{
+							{
+								Name:                   fn.Metadata.Name,
+								Image:                  env.Spec.Runtime.Image,
+								ImagePullPolicy:        apiv1.PullIfNotPresent,
+								TerminationMessagePath: "/dev/termination-log",
+								VolumeMounts: []apiv1.VolumeMount{
+									{
+										Name:      userfunc,
+										MountPath: deploy.sharedMountPath,
+									},
+								},
+								Resources: env.Spec.Resources,
+							},
+							{
+								Name:                   "fetcher",
+								Image:                  deploy.fetcherImg,
+								ImagePullPolicy:        deploy.fetcherImagePullPolicy,
+								TerminationMessagePath: "/dev/termination-log",
+								VolumeMounts: []apiv1.VolumeMount{
+									{
+										Name:      userfunc,
+										MountPath: deploy.sharedMountPath,
+									},
+								},
+								Command: []string{"/fetcher", "-specialize-on-startup",
+									"-fetch-request", string(fetchPayload),
+									"-load-request", string(loadPayload),
+									deploy.sharedMountPath},
+								Env: []apiv1.EnvVar{
+									{
+										Name:  envVersion,
+										Value: strconv.Itoa(env.Spec.Version),
+									},
+								},
+								// TBD Use smaller default resources, for now needed to make HPA work
+								Resources: env.Spec.Resources,
+								ReadinessProbe: &apiv1.Probe{
+									Handler: apiv1.Handler{
+										Exec: &apiv1.ExecAction{
+											Command: []string{"cat", "/tmp/ready"},
+										},
+									},
+									InitialDelaySeconds: 1,
+									PeriodSeconds:       1,
+								},
+							},
+						},
+						ServiceAccountName: "fission-fetcher",
+					},
+				},
+			},
+		}
+		depl, err := deploy.kubernetesClient.ExtensionsV1beta1().Deployments(deploy.namespace).Create(deployment)
+		if err != nil {
+			log.Printf("Error while creating deployment: %v", err)
+			return nil, err
+		}
+
+		for i := 0; i < 120; i++ {
+			latestDepl, err := deploy.kubernetesClient.ExtensionsV1beta1().Deployments(deploy.namespace).Get(depl.Name, metav1.GetOptions{})
+			if err != nil {
+				return nil, err
+			}
+			//TODO check for imagePullerror
+			if latestDepl.Status.ReadyReplicas == replicas {
+				return latestDepl, err
+			}
+			time.Sleep(time.Second)
+		}
+		return nil, errors.New("Failed to create deployment within timeout window")
+	}
+
+	return nil, err
+
+}
+
+func (deploy *NewDeploy) deleteDeployment(ns string, name string) error {
+	deletePropagation := metav1.DeletePropagationForeground
+	err := deploy.kubernetesClient.ExtensionsV1beta1().Deployments(ns).Delete(name, &metav1.DeleteOptions{
+		PropagationPolicy: &deletePropagation,
+	})
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func (deploy *NewDeploy) createOrGetHpa(hpaName string, execStrategy *fission.ExecutionStrategy, depl *v1beta1.Deployment) (*asv1.HorizontalPodAutoscaler, error) {
+
+	minRepl := int32(execStrategy.MinScale)
+	if minRepl == 0 {
+		minRepl = 1
+	}
+	maxRepl := int32(execStrategy.MaxScale)
+	targetCPU := int32(execStrategy.TargetCPUPercent)
+
+	existingHpa, err := deploy.kubernetesClient.AutoscalingV1().HorizontalPodAutoscalers(deploy.namespace).Get(hpaName, metav1.GetOptions{})
+	if err == nil {
+		return existingHpa, err
+	}
+
+	if err != nil && k8s_err.IsNotFound(err) {
+		hpa := asv1.HorizontalPodAutoscaler{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      hpaName,
+				Namespace: deploy.namespace,
+				Labels:    depl.Labels,
+			},
+			Spec: asv1.HorizontalPodAutoscalerSpec{
+				ScaleTargetRef: asv1.CrossVersionObjectReference{
+					Kind:       DeploymentKind,
+					Name:       depl.ObjectMeta.Name,
+					APIVersion: DeploymentVersion,
+				},
+				MinReplicas:                    &minRepl,
+				MaxReplicas:                    maxRepl,
+				TargetCPUUtilizationPercentage: &targetCPU,
+			},
+		}
+
+		cHpa, err := deploy.kubernetesClient.AutoscalingV1().HorizontalPodAutoscalers(deploy.namespace).Create(&hpa)
+		if err != nil {
+			return nil, err
+		}
+		return cHpa, nil
+	}
+
+	return nil, err
+
+}
+
+func (deploy NewDeploy) deleteHpa(ns string, name string) error {
+	err := deploy.kubernetesClient.AutoscalingV1().HorizontalPodAutoscalers(ns).Delete(name, &metav1.DeleteOptions{})
+	return err
+}
+
+func (deploy *NewDeploy) createOrGetSvc(deployLabels map[string]string, svcName string) (*apiv1.Service, error) {
+
+	existingSvc, err := deploy.kubernetesClient.CoreV1().Services(deploy.namespace).Get(svcName, metav1.GetOptions{})
+	if err == nil {
+		return existingSvc, err
+	}
+
+	if err != nil && k8s_err.IsNotFound(err) {
+		service := &apiv1.Service{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:   svcName,
+				Labels: deployLabels,
+			},
+			Spec: apiv1.ServiceSpec{
+				Ports: []apiv1.ServicePort{
+					{
+						Name:       "runtime-env-port",
+						Port:       int32(80),
+						TargetPort: intstr.FromInt(8888),
+					},
+					{
+						Name:       "fetcher-port",
+						Port:       int32(8000),
+						TargetPort: intstr.FromInt(8000),
+					},
+				},
+				Selector: deployLabels,
+				Type:     apiv1.ServiceTypeClusterIP,
+			},
+		}
+
+		svc, err := deploy.kubernetesClient.CoreV1().Services(deploy.namespace).Create(service)
+		if err != nil {
+			return nil, err
+		}
+
+		return svc, nil
+	}
+
+	return nil, err
+}
+
+func (deploy *NewDeploy) deleteSvc(ns string, name string) error {
+	err := deploy.kubernetesClient.CoreV1().Services(ns).Delete(name, &metav1.DeleteOptions{})
+	if err != nil {
+		return err
+	}
+	return nil
+}

executor/newdeploy/newdeploymgr.go

@@ -0,0 +1,357 @@
+/*
+Copyright 2016 The Fission Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package newdeploy
+
+import (
+	"context"
+	"fmt"
+	"log"
+	"os"
+	"time"
+
+	"github.com/fission/fission"
+	"github.com/fission/fission/crd"
+	"github.com/fission/fission/executor/fscache"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/fields"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/pkg/api"
+	apiv1 "k8s.io/client-go/pkg/api/v1"
+	"k8s.io/client-go/rest"
+	k8sCache "k8s.io/client-go/tools/cache"
+)
+
+type (
+	requestType int
+
+	NewDeploy struct {
+		kubernetesClient *kubernetes.Clientset
+		fissionClient    *crd.FissionClient
+		crdClient        *rest.RESTClient
+		instanceID       string
+
+		fetcherImg             string
+		fetcherImagePullPolicy apiv1.PullPolicy
+		namespace              string
+		sharedMountPath        string
+
+		fsCache        *fscache.FunctionServiceCache // cache funcSvc's by function, address and podname
+		requestChannel chan *fnRequest
+
+		functions      []crd.Function
+		funcStore      k8sCache.Store
+		funcController k8sCache.Controller
+	}
+
+	fnRequest struct {
+		reqType         requestType
+		fn              *crd.Function
+		responseChannel chan *fnResponse
+	}
+
+	fnResponse struct {
+		error
+		fSvc *fscache.FuncSvc
+	}
+)
+
+const (
+	FnCreate requestType = iota
+	FnDelete
+	FnUpdate
+)
+
+func MakeNewDeploy(
+	fissionClient *crd.FissionClient,
+	kubernetesClient *kubernetes.Clientset,
+	crdClient *rest.RESTClient,
+	namespace string,
+	fsCache *fscache.FunctionServiceCache,
+	instanceID string,
+) *NewDeploy {
+
+	log.Printf("Creating NewDeploy ExecutorType")
+
+	fetcherImg := os.Getenv("FETCHER_IMAGE")
+	if len(fetcherImg) == 0 {
+		fetcherImg = "fission/fetcher"
+	}
+	fetcherImagePullPolicy := os.Getenv("FETCHER_IMAGE_PULL_POLICY")
+	if len(fetcherImagePullPolicy) == 0 {
+		fetcherImagePullPolicy = "IfNotPresent"
+	}
+
+	nd := &NewDeploy{
+		fissionClient:    fissionClient,
+		kubernetesClient: kubernetesClient,
+		crdClient:        crdClient,
+		instanceID:       instanceID,
+
+		namespace: namespace,
+		fsCache:   fsCache,
+
+		fetcherImg:             fetcherImg,
+		fetcherImagePullPolicy: apiv1.PullIfNotPresent,
+		sharedMountPath:        "/userfunc",
+
+		requestChannel: make(chan *fnRequest),
+	}
+
+	if nd.crdClient != nil {
+		fnStore, fnController := nd.initFuncController()
+		nd.funcStore = fnStore
+		nd.funcController = fnController
+	}
+	go nd.service()
+	return nd
+}
+
+func (deploy *NewDeploy) Run(ctx context.Context) {
+	go deploy.funcController.Run(ctx.Done())
+}
+
+func (deploy *NewDeploy) initFuncController() (k8sCache.Store, k8sCache.Controller) {
+	resyncPeriod := 30 * time.Second
+	listWatch := k8sCache.NewListWatchFromClient(deploy.crdClient, "functions", metav1.NamespaceDefault, fields.Everything())
+	store, controller := k8sCache.NewInformer(listWatch, &crd.Function{}, resyncPeriod, k8sCache.ResourceEventHandlerFuncs{
+		AddFunc: func(obj interface{}) {
+			fn := obj.(*crd.Function)
+			deploy.createFunction(fn)
+		},
+		DeleteFunc: func(obj interface{}) {
+			fn := obj.(*crd.Function)
+			deploy.deleteFunction(fn)
+		},
+		UpdateFunc: func(newObj interface{}, oldObj interface{}) {
+			//TBD
+		},
+	})
+	return store, controller
+}
+
+func (deploy *NewDeploy) service() {
+	for {
+		req := <-deploy.requestChannel
+		switch req.reqType {
+		case FnCreate:
+			fsvc, err := deploy.fnCreate(req.fn)
+			req.responseChannel <- &fnResponse{
+				error: err,
+				fSvc:  fsvc,
+			}
+			continue
+		case FnUpdate:
+			// TBD
+		case FnDelete:
+			_, err := deploy.fnDelete(req.fn)
+			req.responseChannel <- &fnResponse{
+				error: err,
+				fSvc:  nil,
+			}
+			continue
+		}
+	}
+}
+
+func (deploy *NewDeploy) GetFuncSvc(metadata *metav1.ObjectMeta) (*fscache.FuncSvc, error) {
+	c := make(chan *fnResponse)
+	fn, err := deploy.fissionClient.Functions(metadata.Namespace).Get(metadata.Name)
+	if err != nil {
+		return nil, err
+	}
+	deploy.requestChannel <- &fnRequest{
+		fn:              fn,
+		reqType:         FnCreate,
+		responseChannel: c,
+	}
+	resp := <-c
+	if resp.error != nil {
+		return nil, resp.error
+	}
+	return resp.fSvc, nil
+}
+
+func (deploy *NewDeploy) createFunction(fn *crd.Function) {
+	if fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType != fission.ExecutorTypeNewdeploy {
+		return
+	}
+	if fn.Spec.InvokeStrategy.ExecutionStrategy.MinScale <= 0 {
+		return
+	}
+	// Eager creation of function if minScale is greater than 0
+	log.Printf("Eagerly creating newDeploy objects for function")
+	c := make(chan *fnResponse)
+	deploy.requestChannel <- &fnRequest{
+		fn:              fn,
+		reqType:         FnCreate,
+		responseChannel: c,
+	}
+	resp := <-c
+	if resp.error != nil {
+		log.Printf("Error eager creating function: %v", resp.error)
+	}
+}
+
+func (deploy *NewDeploy) deleteFunction(fn *crd.Function) {
+	if fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType == fission.ExecutorTypeNewdeploy {
+		c := make(chan *fnResponse)
+		deploy.requestChannel <- &fnRequest{
+			fn:              fn,
+			reqType:         FnDelete,
+			responseChannel: c,
+		}
+		resp := <-c
+		if resp.error != nil {
+			log.Printf("Error deleing the function: %v", resp.error)
+		}
+	}
+}
+
+func (deploy *NewDeploy) fnCreate(fn *crd.Function) (*fscache.FuncSvc, error) {
+	fsvc, err := deploy.fsCache.GetByFunction(&fn.Metadata)
+	if err == nil {
+		return fsvc, err
+	}
+
+	env, err := deploy.fissionClient.
+		Environments(fn.Spec.Environment.Namespace).
+		Get(fn.Spec.Environment.Name)
+	if err != nil {
+		return fsvc, err
+	}
+
+	objName := deploy.getObjName(fn)
+
+	deployLabels := map[string]string{
+		"environmentName":                 env.Metadata.Name,
+		"environmentUid":                  string(env.Metadata.UID),
+		"functionName":                    fn.Metadata.Name,
+		"functionUid":                     string(fn.Metadata.UID),
+		fission.EXECUTOR_INSTANCEID_LABEL: deploy.instanceID,
+		"executorType":                    fission.ExecutorTypeNewdeploy,
+	}
+
+	depl, err := deploy.createOrGetDeployment(fn, env, objName, deployLabels)
+	if err != nil {
+		log.Printf("Error creating the deployment %v: %v", objName, err)
+		return fsvc, err
+	}
+
+	svc, err := deploy.createOrGetSvc(deployLabels, objName)
+	if err != nil {
+		log.Printf("Error creating the service %v: %v", objName, err)
+		return fsvc, err
+	}
+	svcAddress := svc.Spec.ClusterIP
+
+	hpa, err := deploy.createOrGetHpa(objName, &fn.Spec.InvokeStrategy.ExecutionStrategy, depl)
+	if err != nil {
+		log.Printf("Error creating the HPA %v: %v", objName, err)
+		return fsvc, err
+	}
+
+	kubeObjRefs := []api.ObjectReference{
+		{
+			//obj.TypeMeta.Kind does not work hence this, needs investigationa and a fix
+			Kind:            "deployment",
+			Name:            depl.ObjectMeta.Name,
+			APIVersion:      depl.TypeMeta.APIVersion,
+			Namespace:       depl.ObjectMeta.Namespace,
+			ResourceVersion: depl.ObjectMeta.ResourceVersion,
+			UID:             depl.ObjectMeta.UID,
+		},
+		{
+			Kind:            "service",
+			Name:            svc.ObjectMeta.Name,
+			APIVersion:      svc.TypeMeta.APIVersion,
+			Namespace:       svc.ObjectMeta.Namespace,
+			ResourceVersion: svc.ObjectMeta.ResourceVersion,
+			UID:             svc.ObjectMeta.UID,
+		},
+		{
+			Kind:            "horizontalpodautoscaler",
+			Name:            hpa.ObjectMeta.Name,
+			APIVersion:      hpa.TypeMeta.APIVersion,
+			Namespace:       hpa.ObjectMeta.Namespace,
+			ResourceVersion: hpa.ObjectMeta.ResourceVersion,
+			UID:             hpa.ObjectMeta.UID,
+		},
+	}
+
+	fsvc = &fscache.FuncSvc{
+		Name:              objName,
+		Function:          &fn.Metadata,
+		Environment:       env,
+		Address:           svcAddress,
+		KubernetesObjects: kubeObjRefs,
+		Executor:          fscache.NEWDEPLOY,
+	}
+
+	_, err = deploy.fsCache.Add(*fsvc)
+	if err != nil {
+		log.Printf("Error adding the function to cache: %v", err)
+		return fsvc, err
+	}
+	return fsvc, nil
+}
+
+func (deploy *NewDeploy) fnDelete(fn *crd.Function) (*fscache.FuncSvc, error) {
+
+	var delError error
+
+	fsvc, err := deploy.fsCache.GetByFunction(&fn.Metadata)
+	if err != nil {
+		log.Printf("fsvc not fonud in cache: %v", fn.Metadata)
+		delError = err
+	} else {
+		_, err = deploy.fsCache.DeleteOld(fsvc, time.Second*0)
+		if err != nil {
+			log.Printf("Error deleting the function from cache: %v", fsvc)
+			delError = err
+		}
+	}
+	objName := fsvc.Name
+
+	err = deploy.deleteDeployment(deploy.namespace, objName)
+	if err != nil {
+		log.Printf("Error deleting the deployment: %v", objName)
+		delError = err
+	}
+
+	err = deploy.deleteSvc(deploy.namespace, objName)
+	if err != nil {
+		log.Printf("Error deleting the service: %v", objName)
+		delError = err
+	}
+
+	err = deploy.deleteHpa(deploy.namespace, objName)
+	if err != nil {
+		log.Printf("Error deleting the HPA: %v", objName)
+		delError = err
+	}
+
+	if delError != nil {
+		return nil, delError
+	}
+	return nil, nil
+}
+
+func (deploy *NewDeploy) getObjName(fn *crd.Function) string {
+	return fmt.Sprintf("%v-%v",
+		fn.Metadata.Name,
+		deploy.instanceID)
+}

executor/poolmgr/cleanup.go

@@ -1,147 +0,0 @@
-/*
-Copyright 2016 The Fission Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package poolmgr
-
-import (
-	"log"
-	"time"
-
-	meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/client-go/kubernetes"
-)
-
-// cleanupOldPoolmgrResources looks for resources created by an old
-// poolmgr instance and cleans them up.
-func CleanupOldPoolmgrResources(client *kubernetes.Clientset, namespace string, instanceId string) {
-	go func() {
-		err := cleanup(client, namespace, instanceId)
-		if err != nil {
-			// TODO retry cleanup; logged and ignored for now
-			log.Printf("Failed to cleanup: %v", err)
-		}
-	}()
-}
-
-func cleanup(client *kubernetes.Clientset, namespace string, instanceId string) error {
-	// Deployments are used for idle pools and can be cleaned up
-	// immediately.  (We should "adopt" these instead of creating
-	// a new pool.)
-	err := cleanupDeployments(client, namespace, instanceId)
-	if err != nil {
-		return err
-	}
-	// See K8s #33845 and related bugs: deleting a deployment
-	// through the API doesn't cause the associated ReplicaSet to
-	// be deleted.  (Fixed recently, but we may be running a
-	// version before the fix.)
-	err = cleanupReplicaSets(client, namespace, instanceId)
-	if err != nil {
-		return err
-	}
-
-	// Pods might still be running user functions, so we give them
-	// a few minutes before terminating them.  This time is the
-	// maximum function runtime, plus the time a router might
-	// still route to an old instance, i.e. router cache expiry
-	// time.
-	time.Sleep(6 * time.Minute)
-
-	err = cleanupPods(client, namespace, instanceId)
-	if err != nil {
-		return err
-	}
-
-	err = cleanupServices(client, namespace, instanceId)
-	if err != nil {
-		return err
-	}
-
-	return nil
-}
-
-func cleanupDeployments(client *kubernetes.Clientset, namespace string, instanceId string) error {
-	deploymentList, err := client.ExtensionsV1beta1().Deployments(namespace).List(meta_v1.ListOptions{})
-	if err != nil {
-		return err
-	}
-	for _, dep := range deploymentList.Items {
-		id, ok := dep.ObjectMeta.Labels[POOLMGR_INSTANCEID_LABEL]
-		if ok && id != instanceId {
-			log.Printf("Cleaning up deployment %v", dep.ObjectMeta.Name)
-			err := client.ExtensionsV1beta1().Deployments(namespace).Delete(dep.ObjectMeta.Name, nil)
-			logErr("cleaning up deployment", err)
-			// ignore err
-		}
-	}
-	return nil
-}
-
-func cleanupReplicaSets(client *kubernetes.Clientset, namespace string, instanceId string) error {
-	rsList, err := client.ExtensionsV1beta1().ReplicaSets(namespace).List(meta_v1.ListOptions{})
-	if err != nil {
-		return err
-	}
-	for _, rs := range rsList.Items {
-		id, ok := rs.ObjectMeta.Labels[POOLMGR_INSTANCEID_LABEL]
-		if ok && id != instanceId {
-			log.Printf("Cleaning up replicaset %v", rs.ObjectMeta.Name)
-			err := client.ExtensionsV1beta1().ReplicaSets(namespace).Delete(rs.ObjectMeta.Name, nil)
-			logErr("cleaning up replicaset", err)
-		}
-	}
-	return nil
-}
-
-func cleanupPods(client *kubernetes.Clientset, namespace string, instanceId string) error {
-	podList, err := client.CoreV1().Pods(namespace).List(meta_v1.ListOptions{})
-	if err != nil {
-		return err
-	}
-	for _, pod := range podList.Items {
-		id, ok := pod.ObjectMeta.Labels[POOLMGR_INSTANCEID_LABEL]
-		if ok && id != instanceId {
-			log.Printf("Cleaning up pod %v", pod.ObjectMeta.Name)
-			err := client.CoreV1().Pods(namespace).Delete(pod.ObjectMeta.Name, nil)
-			logErr("cleaning up pod", err)
-			// ignore err
-		}
-	}
-	return nil
-}
-
-func cleanupServices(client *kubernetes.Clientset, namespace string, instanceId string) error {
-	svcList, err := client.CoreV1().Services(namespace).List(meta_v1.ListOptions{})
-	if err != nil {
-		return err
-	}
-	for _, svc := range svcList.Items {
-		id, ok := svc.ObjectMeta.Labels[POOLMGR_INSTANCEID_LABEL]
-		if ok && id != instanceId {
-			log.Printf("Cleaning up svc %v", svc.ObjectMeta.Name)
-			err := client.CoreV1().Services(namespace).Delete(svc.ObjectMeta.Name, nil)
-			logErr("cleaning up svc", err)
-			// ignore err
-		}
-	}
-	return nil
-}
-
-func logErr(msg string, err error) {
-	if err != nil {
-		log.Printf("Error %v: %v", msg, err)
-	}
-}

executor/poolmgr/gp.go

@@ -47,7 +47,6 @@ import (
 	"github.com/fission/fission/executor/fscache"
 )
 
-const POOLMGR_INSTANCEID_LABEL string = "poolmgrInstanceId"
 const POD_PHASE_RUNNING string = "Running"
 
 type (
@@ -144,9 +143,10 @@ func MakeGenericPool(
 
 	// Labels for generic deployment/RS/pods.
 	gp.labelsForPool = map[string]string{
-		"environmentName":        gp.env.Metadata.Name,
-		"environmentUid":         string(gp.env.Metadata.UID),
-		POOLMGR_INSTANCEID_LABEL: gp.instanceId,
+		"environmentName":                 gp.env.Metadata.Name,
+		"environmentUid":                  string(gp.env.Metadata.UID),
+		fission.EXECUTOR_INSTANCEID_LABEL: gp.instanceId,
+		"executorType":                    fission.ExecutorTypePoolmgr,
 	}
 
 	// create the pool
@@ -158,11 +158,6 @@ func MakeGenericPool(
 
 	go gp.choosePodService()
 
-	// Unless specified otherwise, periodically cleanup inactive pods.
-	if env.Spec.AllowedFunctionsPerContainer != fission.AllowedFunctionsPerContainerInfinite {
-		go gp.idlePodReaper()
-	}
-
 	return gp, nil
 }
 
@@ -267,10 +262,10 @@ func (gp *GenericPool) _choosePod(newLabels map[string]string) (*apiv1.Pod, erro
 
 func (gp *GenericPool) labelsForFunction(metadata *metav1.ObjectMeta) map[string]string {
 	return map[string]string{
-		"functionName":           metadata.Name,
-		"functionUid":            string(metadata.UID),
-		"unmanaged":              "true", // this allows us to easily find pods not managed by the deployment
-		POOLMGR_INSTANCEID_LABEL: gp.instanceId,
+		"functionName":                    metadata.Name,
+		"functionUid":                     string(metadata.UID),
+		"unmanaged":                       "true", // this allows us to easily find pods not managed by the deployment
+		fission.EXECUTOR_INSTANCEID_LABEL: gp.instanceId,
 	}
 }
 
@@ -467,6 +462,7 @@ func (gp *GenericPool) createPool() error {
 									MountPath: gp.sharedMountPath,
 								},
 							},
+							Resources: gp.env.Spec.Resources,
 						},
 						{
 							Name:                   "fetcher",
@@ -582,7 +578,7 @@ func (gp *GenericPool) GetFuncSvc(m *metav1.ObjectMeta) (*fscache.FuncSvc, error
 
 	kubeObjRefs := []api.ObjectReference{
 		{
-			Kind:            pod.TypeMeta.Kind,
+			Kind:            "pod",
 			Name:            pod.ObjectMeta.Name,
 			APIVersion:      pod.TypeMeta.APIVersion,
 			Namespace:       pod.ObjectMeta.Namespace,
@@ -592,83 +588,23 @@ func (gp *GenericPool) GetFuncSvc(m *metav1.ObjectMeta) (*fscache.FuncSvc, error
 	}
 
 	fsvc := &fscache.FuncSvc{
+		Name:              pod.ObjectMeta.Name,
 		Function:          m,
 		Environment:       gp.env,
 		Address:           svcHost,
 		KubernetesObjects: kubeObjRefs,
-		Backend:           fscache.POOLMGR,
+		Executor:          fscache.POOLMGR,
 		Ctime:             time.Now(),
 		Atime:             time.Now(),
 	}
 
-	err, _ = gp.fsCache.Add(*fsvc)
+	_, err = gp.fsCache.Add(*fsvc)
 	if err != nil {
 		return nil, err
 	}
 	return fsvc, nil
 }
 
-func (gp *GenericPool) CleanupFunctionService(obj *fscache.FuncSvc) error {
-	// remove ourselves from fsCache (only if we're still old)
-	deleted, err := gp.fsCache.DeleteOld(obj, gp.idlePodReapTime)
-	if err != nil {
-		return err
-	}
-
-	if !deleted {
-		log.Printf("Not deleting %v, in use", obj.Function)
-		return nil
-	}
-
-	for _, kubeobj := range obj.KubernetesObjects {
-		pod, err := gp.kubernetesClient.CoreV1().Pods(gp.namespace).Get(kubeobj.Name, metav1.GetOptions{})
-
-		loggerUrl := fmt.Sprintf("http://%s:1234/v1/log/%s", pod.Spec.NodeName, pod.Name)
-		req, err := http.NewRequest("DELETE", loggerUrl, nil)
-		resp, err := http.DefaultClient.Do(req)
-		if err != nil {
-			log.Printf("Error from %s daemonset logger: %v", pod.Spec.NodeName, err)
-		} else {
-			if resp.StatusCode != 200 {
-				log.Printf("Received not http 200(OK) status from %s daemonset logger: %s", pod.Spec.NodeName, resp.Status)
-			}
-			resp.Body.Close()
-		}
-
-		// delete pod
-		err = gp.kubernetesClient.CoreV1().Pods(gp.namespace).Delete(kubeobj.Name, nil)
-		if err != nil {
-			return err
-		}
-		if err != nil {
-			return err
-		}
-	}
-
-	return nil
-}
-
-func (gp *GenericPool) idlePodReaper() {
-	for {
-		time.Sleep(time.Minute)
-		funcSvcs, err := gp.fsCache.ListOld(&gp.env.Metadata, gp.idlePodReapTime)
-		if err != nil {
-			log.Printf("Error reaping idle pods: %v", err)
-			continue
-		}
-		for _, obj := range funcSvcs {
-			err := gp.CleanupFunctionService(obj)
-			if err != nil {
-				log.Printf("Error deleting Kubernetes objects for fsvc '%v': %v", obj, err)
-				log.Printf("Object Name| Object Kind | Object Space")
-				for _, kubeobj := range obj.KubernetesObjects {
-					log.Printf("%v | %v | %v", kubeobj.Name, kubeobj.Kind, kubeobj.Namespace)
-				}
-			}
-		}
-	}
-}
-
 // destroys the pool -- the deployment, replicaset and pods
 func (gp *GenericPool) destroy() error {
 	// Destroy deployment

executor/poolmgr/gpm.go

@@ -89,7 +89,7 @@ func (gpm *GenericPoolManager) service() {
 			var err error
 			pool, ok := gpm.pools[crd.CacheKey(&req.env.Metadata)]
 			if !ok {
-				var poolSize int32 = 3 // TODO configurable/autoscalable
+				var poolSize = int32(req.env.Spec.Poolsize)
 				switch req.env.Spec.AllowedFunctionsPerContainer {
 				case fission.AllowedFunctionsPerContainerInfinite:
 					poolSize = 1
@@ -107,13 +107,17 @@ func (gpm *GenericPoolManager) service() {
 			req.responseChannel <- &response{pool: pool}
 		case CLEANUP_POOLS:
 			latestEnvSet := make(map[string]bool)
+			latestEnvPoolsize := make(map[string]int)
 			for _, env := range req.envList {
 				latestEnvSet[crd.CacheKey(&env.Metadata)] = true
+				latestEnvPoolsize[crd.CacheKey(&env.Metadata)] = env.Spec.Poolsize
 			}
 			for key, pool := range gpm.pools {
 				_, ok := latestEnvSet[key]
-				if !ok {
-					// Env no longer exists -- remove our cache
+				poolsize := latestEnvPoolsize[key]
+				if !ok || poolsize == 0 {
+					// Env no longer exists or pool size changed to zero
+
 					log.Printf("Destroying generic pool for environment [%v]", key)
 					delete(gpm.pools, key)
 
@@ -160,9 +164,13 @@ func (gpm *GenericPoolManager) eagerPoolCreator() {
 		// to keep these eagerly created pools smaller than the ones created when there are
 		// actual function calls.
 		for i := range envs.Items {
-			_, err := gpm.GetPool(&envs.Items[i])
-			if err != nil {
-				log.Printf("eager-create pool failed: %v", err)
+			env := envs.Items[i]
+			// Create pool only if poolsize greater than zero
+			if env.Spec.Poolsize > 0 {
+				_, err := gpm.GetPool(&envs.Items[i])
+				if err != nil {
+					log.Printf("eager-create pool failed: %v", err)
+				}
 			}
 		}