Newdeploy backend (#387)
A newdeploy backend which uses new deployment to serve requests. This is the second phase of #193 and builds on top of changes in #384 . * Executor layer added on top of pool manager * Removed the external server for executor * Minor changes to keep existing semantics as much possible * Separating the executor vs. poolmgr backend functionality and associated data members * Executor logic separated from Poolmgr backend completely, placeholder for new backend * Changed references to poolmgr in tests * Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package * Rebased from master and changed references to tpr -> crd * Executor layer added on top of pool manager * Executor logic separated from Poolmgr backend completely, placeholder for new backend * Changed podName to a generic objectReference in fscache (#391) Changed podName to a generic objectReference in function service cache implementation. * Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package * Rebased from master and changed references to tpr -> crd * Merged from master with latest changes * Executor layer added on top of pool manager * Removed the external server for executor * Minor changes to keep existing semantics as much possible * Separating the executor vs. poolmgr backend functionality and associated data members * Executor logic separated from Poolmgr backend completely, placeholder for new backend * Changed references to poolmgr in tests * update compiling.md to use helm * Compile instructions: changed pullPolicy to IfNotPresent (#378) Containers will get stuck in ErrImagePull/ImagePullBackOff state otherwise * Moved poolmgr to it's package, as a side effect moved Cache to its's package (was causing cyclical dependency) and had to make some data structures exposed outside package * Fetcher called when pod is created for newDeploy backend but also supports older way, this is WIP and still needs pod specialization and creating & exposing a service so the URL can be hit by end user * WIP Specializing the POD as part of startup along with fetching * Working specialization of a new deployment. Needs some work on caching, cleanup etc. * Switched to service based address instead of POD address * Minor formating issue fixed * Added logging to pods and a readiness check, the readiness check is flaky though ATM * Fixed some rebase issues that were failing build * Better names for K8S objects and methods * Switched usage of FuncSvc in backends from pod to api.ObjectReference * Adding retry to fetcher request, for now just using default retry client which might need tweaking in future * Switching to plain old retry, some issue in getting retryablehttp with glide import * Removed stale executor service & deployment from previous merge * Addressed review comments, still testing some areas * Added types in FunctionSpec * Resolved conflicts due to merge from executor_abstraction branch * Added backend type on EnvironmentSpec along with operations for create/list/update, the pools are created/destroyed based on change in backend type * Backend from types and a minor err return issue fixed * Draft version of CPU and memory parameters added to environment * Added resourceReq to newDeploy, though it has some issues * Issue with resourceName fixed, now newdeploy pods also pick up resources from the environment config * Adding scale params, removing validation on CPU params for now * Fixed a formatting issue * Checking if slight more delay helps in the test which is currently failing for internal routes * The resourceList newly added in Env can not be compared by compiler, hence must use breakdown comparison instead * Added strategy selection on client side * Added caching, informers, delete operations for newdeploy backend functions * Deleted a stale directory * A simple HPA based on scale parameters, testing still WIP * Fixed a small issue in delete function, added HPA delete too when deleting a function * Previous merge missed the pkg flag for update fn command somehow, fixed that * Fixed comments from review * Changed poolmgr cleanup to be generic cleanup and moved to executor, added instanceID labels to newdeploy so that cleanup works * Moved instanceIdLabel to types to avoid cyclic dependency * More review fixes * Tweaking sleep to see results * If user does not provide poolsize, then it should not default to zero * Switched to naming convention for now, fixed default poolsize if not provided * Changed error return behaviour in delete fn, also changed cleanup to look based on obj type though support for additional type will need more work * Changed check location so avoid false logging * Test for newdeploy backend * Adding tests for poolmgr backend * Fixed an issue with glide dependency version, already fixed in master * Added instanceId for NewDeploy, Initial cleanup now cleans older objects of newdeploy backend, removed eagercreate flag and instead using minScale to drive eager creation * Moved cleanup to executor layer with cleanup for newDeploy backend, changes to use the new Cache impl * Cleaning up pod & rs along with deployment for newdeploy backend * Enhanced fn and env listing to show min/maxscale and resuorces respectively * Added conditional heapster deployment and fixed a small issue with resources for fetcher container in function pod * Addressed review comments from previous change * Addressed some more review comments - majorly create only on NotFoundError * Added TargetCPU as an input for scaling * Bumped target CPU to be greater than 0 and added a default value * Min replicas should be 1 even if the minScale is 0 when creating deployment * Changed name from 'backend' to executorType, added additional test for minscale 0 case, changed TargetCPU to TargetCPUPercent
This commit is contained in:
Vishal
@@ -1,16 +1,35 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/fission/fission"
|
||||
"github.com/fission/fission/environments/fetcher"
|
||||
)
|
||||
|
||||
// Usage: fetcher <shared volume path>
|
||||
func main() {
|
||||
dir := os.Args[1]
|
||||
flag.Usage = fetcherUsage
|
||||
fetchPayload := flag.String("fetch-request", "", "JSON Payload for fetch request")
|
||||
loadPayload := flag.String("load-request", "", "JSON payload for Load request")
|
||||
specializeOnStart := flag.Bool("specialize-on-startup", false, "Flag to activate specialize process at pod starup")
|
||||
flag.Parse()
|
||||
if flag.NArg() == 0 {
|
||||
flag.Usage()
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
dir := flag.Arg(0)
|
||||
if _, err := os.Stat(dir); err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
err = os.MkdirAll(dir, os.ModeDir|0700)
|
||||
@@ -19,7 +38,13 @@ func main() {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fetcher := fetcher.MakeFetcher(dir)
|
||||
|
||||
if *specializeOnStart {
|
||||
specializePod(fetcher, fetchPayload, loadPayload)
|
||||
}
|
||||
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/", fetcher.FetchHandler)
|
||||
mux.HandleFunc("/upload", fetcher.UploadHandler)
|
||||
@@ -28,3 +53,80 @@ func main() {
|
||||
})
|
||||
http.ListenAndServe(":8000", mux)
|
||||
}
|
||||
|
||||
func fetcherUsage() {
|
||||
fmt.Printf("Usage: fetcher [-specialize-on-startup] [-fetch-request <json>] [-load-request <json>] <shared volume path> \n")
|
||||
}
|
||||
|
||||
func specializePod(f *fetcher.Fetcher, fetchPayload *string, loadPayload *string) {
|
||||
// Fetch code
|
||||
var fetchReq fetcher.FetchRequest
|
||||
err := json.Unmarshal([]byte(*fetchPayload), &fetchReq)
|
||||
if err != nil {
|
||||
log.Fatalf("Error parsing fetch request: %v", err)
|
||||
}
|
||||
_, err = f.Fetch(fetchReq)
|
||||
if err != nil {
|
||||
log.Fatalf("Error fetching: %v", err)
|
||||
}
|
||||
|
||||
// Specialize the pod
|
||||
|
||||
envVersion, err := strconv.Atoi(os.Getenv("ENV_VERSION"))
|
||||
if err != nil {
|
||||
log.Fatalf("Error parsing environment version %v, error: %v", os.Getenv("ENV_VERSION"), err)
|
||||
}
|
||||
|
||||
maxRetries := 30
|
||||
var contentType string
|
||||
var specializeURL string
|
||||
var reader *bytes.Reader
|
||||
|
||||
if envVersion == 2 {
|
||||
contentType = "application/json"
|
||||
specializeURL = "http://localhost:8888/v2/specialize"
|
||||
reader = bytes.NewReader([]byte(*loadPayload))
|
||||
} else {
|
||||
contentType = "text/plain"
|
||||
specializeURL = "http://localhost:8888/specialize"
|
||||
reader = bytes.NewReader([]byte{})
|
||||
}
|
||||
|
||||
for i := 0; i < maxRetries; i++ {
|
||||
resp, err := http.Post(specializeURL, contentType, reader)
|
||||
if err == nil && resp.StatusCode < 300 {
|
||||
// Success
|
||||
resp.Body.Close()
|
||||
//On Success creates a file which is used as a readiness probe by Kubernetes for this container/pod
|
||||
file, err := os.OpenFile("/tmp/ready", os.O_RDONLY|os.O_CREATE, 0666)
|
||||
if err != nil {
|
||||
log.Fatalf("Error creating readiness file: %v", err)
|
||||
}
|
||||
err = file.Close()
|
||||
if err != nil {
|
||||
log.Fatalf("Error closing readiness file: %v", err)
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
// Only retry for the specific case of a connection error.
|
||||
if urlErr, ok := err.(*url.Error); ok {
|
||||
if netErr, ok := urlErr.Err.(*net.OpError); ok {
|
||||
if netErr.Op == "dial" {
|
||||
if i < maxRetries-1 {
|
||||
time.Sleep(500 * time.Duration(2*i) * time.Millisecond)
|
||||
log.Printf("Error connecting to pod (%v), retrying", netErr)
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err == nil {
|
||||
err = fission.MakeErrorFromHTTP(resp)
|
||||
}
|
||||
log.Printf("Failed to specialize pod: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -157,6 +157,19 @@ func (fetcher *Fetcher) FetchHandler(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
log.Printf("fetcher received fetch request and started downloading: %v", req)
|
||||
|
||||
code, err := fetcher.Fetch(req)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), code)
|
||||
return
|
||||
}
|
||||
|
||||
// all done
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
|
||||
// Fetch takes FetchRequest and makes the fetch call
|
||||
// It returns the HTTP code and error if any
|
||||
func (fetcher *Fetcher) Fetch(req FetchRequest) (int, error) {
|
||||
tmpFile := req.Filename + ".tmp"
|
||||
tmpPath := filepath.Join(fetcher.sharedVolumePath, tmpFile)
|
||||
|
||||
@@ -166,8 +179,7 @@ func (fetcher *Fetcher) FetchHandler(w http.ResponseWriter, r *http.Request) {
|
||||
if err != nil {
|
||||
e := fmt.Sprintf("Failed to download url %v: %v", req.Url, err)
|
||||
log.Printf(e)
|
||||
http.Error(w, e, 400)
|
||||
return
|
||||
return 400, errors.New(e)
|
||||
}
|
||||
} else {
|
||||
// get pkg
|
||||
@@ -175,8 +187,7 @@ func (fetcher *Fetcher) FetchHandler(w http.ResponseWriter, r *http.Request) {
|
||||
if err != nil {
|
||||
e := fmt.Sprintf("Failed to get package: %v", err)
|
||||
log.Printf(e)
|
||||
http.Error(w, e, 500)
|
||||
return
|
||||
return 500, errors.New(e)
|
||||
}
|
||||
|
||||
var archive *fission.Archive
|
||||
@@ -185,7 +196,6 @@ func (fetcher *Fetcher) FetchHandler(w http.ResponseWriter, r *http.Request) {
|
||||
} else if req.FetchType == FETCH_DEPLOYMENT {
|
||||
archive = &pkg.Spec.Deployment
|
||||
}
|
||||
|
||||
// get package data as literal or by url
|
||||
if len(archive.Literal) > 0 {
|
||||
// write pkg.Literal into tmpPath
|
||||
@@ -193,26 +203,22 @@ func (fetcher *Fetcher) FetchHandler(w http.ResponseWriter, r *http.Request) {
|
||||
if err != nil {
|
||||
e := fmt.Sprintf("Failed to write file %v: %v", tmpPath, err)
|
||||
log.Printf(e)
|
||||
http.Error(w, e, 500)
|
||||
return
|
||||
return 500, errors.New(e)
|
||||
}
|
||||
} else {
|
||||
// download and verify
|
||||
|
||||
err = downloadUrl(archive.URL, tmpPath)
|
||||
if err != nil {
|
||||
e := fmt.Sprintf("Failed to download url %v: %v", req.Url, err)
|
||||
log.Printf(e)
|
||||
http.Error(w, e, 400)
|
||||
return
|
||||
return 400, errors.New(e)
|
||||
}
|
||||
|
||||
err = verifyChecksum(tmpPath, &archive.Checksum)
|
||||
if err != nil {
|
||||
e := fmt.Sprintf("Failed to verify checksum: %v", err)
|
||||
log.Printf(e)
|
||||
http.Error(w, e, 400)
|
||||
return
|
||||
return 400, errors.New(e)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -221,26 +227,22 @@ func (fetcher *Fetcher) FetchHandler(w http.ResponseWriter, r *http.Request) {
|
||||
if archiver.Zip.Match(tmpPath) {
|
||||
// unarchive tmp file to a tmp unarchive path
|
||||
tmpUnarchivePath := filepath.Join(fetcher.sharedVolumePath, uuid.NewV4().String())
|
||||
err = fetcher.unarchive(tmpPath, tmpUnarchivePath)
|
||||
err := fetcher.unarchive(tmpPath, tmpUnarchivePath)
|
||||
if err != nil {
|
||||
log.Println(err.Error())
|
||||
http.Error(w, err.Error(), 500)
|
||||
return
|
||||
return 500, err
|
||||
}
|
||||
tmpPath = tmpUnarchivePath
|
||||
}
|
||||
|
||||
// move tmp file to requested filename
|
||||
err = fetcher.rename(tmpPath, filepath.Join(fetcher.sharedVolumePath, req.Filename))
|
||||
err := fetcher.rename(tmpPath, filepath.Join(fetcher.sharedVolumePath, req.Filename))
|
||||
if err != nil {
|
||||
log.Println(err.Error())
|
||||
http.Error(w, err.Error(), 500)
|
||||
return
|
||||
return 500, err
|
||||
}
|
||||
|
||||
log.Printf("Completed fetch request")
|
||||
// all done
|
||||
w.WriteHeader(http.StatusOK)
|
||||
log.Printf("Successfully placed at %v", filepath.Join(fetcher.sharedVolumePath, req.Filename))
|
||||
return 200, nil
|
||||
}
|
||||
|
||||
func (fetcher *Fetcher) UploadHandler(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
Reference in New Issue
Block a user