Newdeploy backend (#387)

A newdeploy backend which uses new deployment to serve requests. This is the second phase of #193 and builds on top of changes in #384 .

* Executor layer added on top of pool manager

* Removed the external server for executor

* Minor changes to keep existing semantics as much possible

* Separating the executor vs. poolmgr backend functionality and associated data members

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed references to poolmgr in tests

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Rebased from master and changed references to tpr -> crd

* Executor layer added on top of pool manager

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed podName to a generic objectReference in fscache (#391)

Changed podName to a generic objectReference in function service cache implementation.

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Rebased from master and changed references to tpr -> crd

* Merged from master with latest changes

* Executor layer added on top of pool manager

* Removed the external server for executor

* Minor changes to keep existing semantics as much possible

* Separating the executor vs. poolmgr backend functionality and associated data members

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed references to poolmgr in tests

* update compiling.md to use helm

* Compile instructions: changed pullPolicy to IfNotPresent (#378)

Containers will get stuck in ErrImagePull/ImagePullBackOff state otherwise

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Fetcher called when pod is created for newDeploy backend but also supports older way, this is WIP and still needs pod specialization and creating & exposing a service so the URL can be hit by end user

* WIP Specializing the POD as part of startup along with fetching

* Working specialization of a new deployment. Needs some work on caching, cleanup etc.

* Switched to service based address instead of POD address

* Minor formating issue fixed

* Added logging to pods and a readiness check, the readiness check is flaky though ATM

* Fixed some rebase issues that were failing build

* Better names for K8S objects and methods

* Switched usage of FuncSvc in backends from pod to api.ObjectReference

* Adding retry to fetcher request, for now just using default retry client which might need tweaking in future

* Switching to plain old retry, some issue in getting retryablehttp with glide import

* Removed stale executor service & deployment from previous merge

* Addressed review comments, still testing some areas

* Added types in FunctionSpec

* Resolved conflicts due to merge from executor_abstraction branch

* Added backend type on EnvironmentSpec along with operations for create/list/update, the pools are created/destroyed based on change in backend type

* Backend from types and a minor err return issue fixed

* Draft version of CPU and memory parameters added to environment

* Added resourceReq to newDeploy, though it has some issues

* Issue with resourceName fixed, now newdeploy pods also pick up resources from the environment config

* Adding scale params, removing validation on CPU params for now

* Fixed a formatting issue

* Checking if slight more delay helps in the test which is currently failing for internal routes

* The resourceList newly added in Env can not be compared by compiler, hence must use breakdown comparison instead

* Added strategy selection on client side

* Added caching, informers, delete operations for newdeploy backend functions

* Deleted a stale directory

* A simple HPA based on scale parameters, testing still WIP

* Fixed a small issue in delete function, added HPA delete too when deleting a function

* Previous merge missed the pkg flag for update fn command somehow, fixed that

* Fixed comments from review

* Changed poolmgr cleanup to be generic cleanup and moved to executor, added instanceID labels to newdeploy so that cleanup works

* Moved instanceIdLabel to types to avoid cyclic dependency

* More review fixes

* Tweaking sleep to see results

* If user does not provide poolsize, then it should not default to zero

* Switched to naming convention for now, fixed default poolsize if not provided

* Changed error return behaviour in delete fn, also changed cleanup to look based on obj type though support for additional type will need more work

* Changed check location so avoid false logging

* Test for newdeploy backend

* Adding tests for poolmgr backend

* Fixed an issue with glide dependency version, already fixed in master

* Added instanceId for NewDeploy, Initial cleanup now cleans older objects of newdeploy backend, removed eagercreate flag and instead using minScale to drive eager creation

* Moved cleanup to executor layer with cleanup for newDeploy backend, changes to use the new Cache impl

* Cleaning up pod & rs along with deployment for newdeploy backend

* Enhanced fn and env listing to show min/maxscale and resuorces respectively

* Added conditional heapster deployment and fixed a small issue with resources for fetcher container in function pod

* Addressed review comments from previous change

* Addressed some more review comments - majorly create only on NotFoundError

* Added TargetCPU as an input for scaling

* Bumped target CPU to be greater than 0 and added a default value

* Min replicas should be 1 even if the minScale is 0 when creating deployment

* Changed name from 'backend' to executorType, added additional test for minscale 0 case, changed TargetCPU to TargetCPUPercent

executor/executor.go

@@ -18,22 +18,25 @@ package executor
 
 import (
 	"log"
-	"os"
+	"strings"
 	"sync"
 	"time"
 
 	"github.com/dchest/uniuri"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
+	"github.com/fission/fission"
 	"github.com/fission/fission/cache"
 	"github.com/fission/fission/crd"
 	"github.com/fission/fission/executor/fscache"
+	"github.com/fission/fission/executor/newdeploy"
 	"github.com/fission/fission/executor/poolmgr"
 )
 
 type (
 	Executor struct {
 		gpm           *poolmgr.GenericPoolManager
+		ndm           *newdeploy.NewDeploy
 		functionEnv   *cache.Cache
 		fissionClient *crd.FissionClient
 		fsCache       *fscache.FunctionServiceCache
@@ -52,9 +55,10 @@ type (
 	}
 )
 
-func MakeExecutor(gpm *poolmgr.GenericPoolManager, fissionClient *crd.FissionClient, fsCache *fscache.FunctionServiceCache) *Executor {
+func MakeExecutor(gpm *poolmgr.GenericPoolManager, ndm *newdeploy.NewDeploy, fissionClient *crd.FissionClient, fsCache *fscache.FunctionServiceCache) *Executor {
 	executor := &Executor{
 		gpm:           gpm,
+		ndm:           ndm,
 		functionEnv:   cache.MakeCache(10*time.Second, 0),
 		fissionClient: fissionClient,
 		fsCache:       fsCache,
@@ -114,18 +118,27 @@ func (executor *Executor) serveCreateFuncServices() {
 	}
 }
 
-func (executor *Executor) createServiceForFunction(m *metav1.ObjectMeta) (*fscache.FuncSvc, error) {
-	log.Printf("[%v] No cached function service found, creating one", m.Name)
+func (executor *Executor) createServiceForFunction(meta *metav1.ObjectMeta) (*fscache.FuncSvc, error) {
+	log.Printf("[%v] No cached function service found, creating one", meta.Name)
 
-	env, err := executor.getFunctionEnv(m)
+	// from Func -> get Env
+	log.Printf("[%v] getting environment for function", meta.Name)
+	env, err := executor.getFunctionEnv(meta)
 	if err != nil {
 		return nil, err
 	}
-	// Appropriate backend handles the service creation
-	backend := os.Getenv("EXECUTOR_BACKEND")
-	switch backend {
-	case "DEPLOY":
-		return nil, nil
+
+	fn, err := executor.fissionClient.
+		Functions(meta.Namespace).
+		Get(meta.Name)
+	if err != nil {
+		return nil, err
+	}
+
+	switch fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType {
+	case fission.ExecutorTypeNewdeploy:
+		fs, err := executor.ndm.GetFuncSvc(meta)
+		return fs, err
 	default:
 		pool, err := executor.gpm.GetPool(env)
 		if err != nil {
@@ -133,12 +146,9 @@ func (executor *Executor) createServiceForFunction(m *metav1.ObjectMeta) (*fscac
 		}
 		// from GenericPool -> get one function container
 		// (this also adds to the cache)
-		log.Printf("[%v] getting function service from pool", m.Name)
-		fsvc, err := pool.GetFuncSvc(m)
-		if err != nil {
-			return nil, err
-		}
-		return fsvc, nil
+		log.Printf("[%v] getting function service from pool", meta.Name)
+		fsvc, err := pool.GetFuncSvc(meta)
+		return fsvc, err
 	}
 }
 
@@ -171,24 +181,31 @@ func (executor *Executor) getFunctionEnv(m *metav1.ObjectMeta) (*crd.Environment
 	return env, nil
 }
 
-// StartExecutor Starts executor and the backend components that executor uses such as Poolmgr,
-// deploymgr and potential future backends
+// StartExecutor Starts executor and the executor components such as Poolmgr,
+// deploymgr and potential future executor types
 func StartExecutor(fissionNamespace string, functionNamespace string, port int) error {
 	fissionClient, kubernetesClient, _, err := crd.MakeFissionClient()
+	restClient := fissionClient.GetCrdClient()
 	if err != nil {
 		log.Printf("Failed to get kubernetes client: %v", err)
 		return err
 	}
 
-	instanceID := uniuri.NewLen(8)
-	poolmgr.CleanupOldPoolmgrResources(kubernetesClient, functionNamespace, instanceID)
-
 	fsCache := fscache.MakeFunctionServiceCache()
+
+	poolID := strings.ToLower(uniuri.NewLen(8))
+	cleanupObjects(kubernetesClient, functionNamespace, poolID)
+	go idleObjectReaper(kubernetesClient, fissionClient, fsCache, time.Minute*2)
 	gpm := poolmgr.MakeGenericPoolManager(
 		fissionClient, kubernetesClient, fissionNamespace,
-		functionNamespace, fsCache, instanceID)
+		functionNamespace, fsCache, poolID)
+
+	ndm := newdeploy.MakeNewDeploy(
+		fissionClient, kubernetesClient, restClient,
+		functionNamespace, fsCache, poolID)
+
+	api := MakeExecutor(gpm, ndm, fissionClient, fsCache)
 
-	api := MakeExecutor(gpm, fissionClient, fsCache)
 	go api.Serve(port)
 
 	return nil