Newdeploy backend (#387)

A newdeploy backend which uses new deployment to serve requests. This is the second phase of #193 and builds on top of changes in #384 .

* Executor layer added on top of pool manager

* Removed the external server for executor

* Minor changes to keep existing semantics as much possible

* Separating the executor vs. poolmgr backend functionality and associated data members

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed references to poolmgr in tests

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Rebased from master and changed references to tpr -> crd

* Executor layer added on top of pool manager

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed podName to a generic objectReference in fscache (#391)

Changed podName to a generic objectReference in function service cache implementation.

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Rebased from master and changed references to tpr -> crd

* Merged from master with latest changes

* Executor layer added on top of pool manager

* Removed the external server for executor

* Minor changes to keep existing semantics as much possible

* Separating the executor vs. poolmgr backend functionality and associated data members

* Executor logic separated from Poolmgr backend completely, placeholder for new backend

* Changed references to poolmgr in tests

* update compiling.md to use helm

* Compile instructions: changed pullPolicy to IfNotPresent (#378)

Containers will get stuck in ErrImagePull/ImagePullBackOff state otherwise

* Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package

* Fetcher called when pod is created for newDeploy backend but also supports older way, this is WIP and still needs pod specialization and creating & exposing a service so the URL can be hit by end user

* WIP Specializing the POD as part of startup along with fetching

* Working specialization of a new deployment. Needs some work on caching, cleanup etc.

* Switched to service based address instead of POD address

* Minor formating issue fixed

* Added logging to pods and a readiness check, the readiness check is flaky though ATM

* Fixed some rebase issues that were failing build

* Better names for K8S objects and methods

* Switched usage of FuncSvc in backends from pod to api.ObjectReference

* Adding retry to fetcher request, for now just using default retry client which might need tweaking in future

* Switching to plain old retry, some issue in getting retryablehttp with glide import

* Removed stale executor service & deployment from previous merge

* Addressed review comments, still testing some areas

* Added types in FunctionSpec

* Resolved conflicts due to merge from executor_abstraction branch

* Added backend type on EnvironmentSpec along with operations for create/list/update, the pools are created/destroyed based on change in backend type

* Backend from types and a minor err return issue fixed

* Draft version of CPU and memory parameters added to environment

* Added resourceReq to newDeploy, though it has some issues

* Issue with resourceName fixed, now newdeploy pods also pick up resources from the environment config

* Adding scale params, removing validation on CPU params for now

* Fixed a formatting issue

* Checking if slight more delay helps in the test which is currently failing for internal routes

* The resourceList newly added in Env can not be compared by compiler, hence must use breakdown comparison instead

* Added strategy selection on client side

* Added caching, informers, delete operations for newdeploy backend functions

* Deleted a stale directory

* A simple HPA based on scale parameters, testing still WIP

* Fixed a small issue in delete function, added HPA delete too when deleting a function

* Previous merge missed the pkg flag for update fn command somehow, fixed that

* Fixed comments from review

* Changed poolmgr cleanup to be generic cleanup and moved to executor, added instanceID labels to newdeploy so that cleanup works

* Moved instanceIdLabel to types to avoid cyclic dependency

* More review fixes

* Tweaking sleep to see results

* If user does not provide poolsize, then it should not default to zero

* Switched to naming convention for now, fixed default poolsize if not provided

* Changed error return behaviour in delete fn, also changed cleanup to look based on obj type though support for additional type will need more work

* Changed check location so avoid false logging

* Test for newdeploy backend

* Adding tests for poolmgr backend

* Fixed an issue with glide dependency version, already fixed in master

* Added instanceId for NewDeploy, Initial cleanup now cleans older objects of newdeploy backend, removed eagercreate flag and instead using minScale to drive eager creation

* Moved cleanup to executor layer with cleanup for newDeploy backend, changes to use the new Cache impl

* Cleaning up pod & rs along with deployment for newdeploy backend

* Enhanced fn and env listing to show min/maxscale and resuorces respectively

* Added conditional heapster deployment and fixed a small issue with resources for fetcher container in function pod

* Addressed review comments from previous change

* Addressed some more review comments - majorly create only on NotFoundError

* Added TargetCPU as an input for scaling

* Bumped target CPU to be greater than 0 and added a default value

* Min replicas should be 1 even if the minScale is 0 when creating deployment

* Changed name from 'backend' to executorType, added additional test for minscale 0 case, changed TargetCPU to TargetCPUPercent

environments/fetcher/cmd/main.go

@@ -1,16 +1,35 @@
 package main
 
 import (
+	"bytes"
+	"encoding/json"
+	"flag"
+	"fmt"
 	"log"
+	"net"
 	"net/http"
+	"net/url"
 	"os"
+	"strconv"
+	"time"
 
+	"github.com/fission/fission"
 	"github.com/fission/fission/environments/fetcher"
 )
 
 // Usage: fetcher <shared volume path>
 func main() {
-	dir := os.Args[1]
+	flag.Usage = fetcherUsage
+	fetchPayload := flag.String("fetch-request", "", "JSON Payload for fetch request")
+	loadPayload := flag.String("load-request", "", "JSON payload for Load request")
+	specializeOnStart := flag.Bool("specialize-on-startup", false, "Flag to activate specialize process at pod starup")
+	flag.Parse()
+	if flag.NArg() == 0 {
+		flag.Usage()
+		os.Exit(1)
+	}
+
+	dir := flag.Arg(0)
 	if _, err := os.Stat(dir); err != nil {
 		if os.IsNotExist(err) {
 			err = os.MkdirAll(dir, os.ModeDir|0700)
@@ -19,7 +38,13 @@ func main() {
 			}
 		}
 	}
+
 	fetcher := fetcher.MakeFetcher(dir)
+
+	if *specializeOnStart {
+		specializePod(fetcher, fetchPayload, loadPayload)
+	}
+
 	mux := http.NewServeMux()
 	mux.HandleFunc("/", fetcher.FetchHandler)
 	mux.HandleFunc("/upload", fetcher.UploadHandler)
@@ -28,3 +53,80 @@ func main() {
 	})
 	http.ListenAndServe(":8000", mux)
 }
+
+func fetcherUsage() {
+	fmt.Printf("Usage: fetcher [-specialize-on-startup] [-fetch-request <json>] [-load-request <json>] <shared volume path> \n")
+}
+
+func specializePod(f *fetcher.Fetcher, fetchPayload *string, loadPayload *string) {
+	// Fetch code
+	var fetchReq fetcher.FetchRequest
+	err := json.Unmarshal([]byte(*fetchPayload), &fetchReq)
+	if err != nil {
+		log.Fatalf("Error parsing fetch request: %v", err)
+	}
+	_, err = f.Fetch(fetchReq)
+	if err != nil {
+		log.Fatalf("Error fetching: %v", err)
+	}
+
+	// Specialize the pod
+
+	envVersion, err := strconv.Atoi(os.Getenv("ENV_VERSION"))
+	if err != nil {
+		log.Fatalf("Error parsing environment version %v, error: %v", os.Getenv("ENV_VERSION"), err)
+	}
+
+	maxRetries := 30
+	var contentType string
+	var specializeURL string
+	var reader *bytes.Reader
+
+	if envVersion == 2 {
+		contentType = "application/json"
+		specializeURL = "http://localhost:8888/v2/specialize"
+		reader = bytes.NewReader([]byte(*loadPayload))
+	} else {
+		contentType = "text/plain"
+		specializeURL = "http://localhost:8888/specialize"
+		reader = bytes.NewReader([]byte{})
+	}
+
+	for i := 0; i < maxRetries; i++ {
+		resp, err := http.Post(specializeURL, contentType, reader)
+		if err == nil && resp.StatusCode < 300 {
+			// Success
+			resp.Body.Close()
+			//On Success creates a file which is used as a readiness probe by Kubernetes for this container/pod
+			file, err := os.OpenFile("/tmp/ready", os.O_RDONLY|os.O_CREATE, 0666)
+			if err != nil {
+				log.Fatalf("Error creating readiness file: %v", err)
+			}
+			err = file.Close()
+			if err != nil {
+				log.Fatalf("Error closing readiness file: %v", err)
+			}
+			break
+		}
+
+		// Only retry for the specific case of a connection error.
+		if urlErr, ok := err.(*url.Error); ok {
+			if netErr, ok := urlErr.Err.(*net.OpError); ok {
+				if netErr.Op == "dial" {
+					if i < maxRetries-1 {
+						time.Sleep(500 * time.Duration(2*i) * time.Millisecond)
+						log.Printf("Error connecting to pod (%v), retrying", netErr)
+						continue
+					}
+				}
+			}
+		}
+
+		if err == nil {
+			err = fission.MakeErrorFromHTTP(resp)
+		}
+		log.Printf("Failed to specialize pod: %v", err)
+		return
+	}
+
+}