Fission 1.7.0

Add CleanupOldExecutorObjects to executor type interface in order
to let an executor type manages how to clean up the resources it created.

CHANGELOG.md

@@ -1,35 +1,134 @@
-# 1.6.0
+# 1.7.0-rc.2
 [Documentation](https://docs.fission.io/)
-## Downloads for 1.6.0
+## Downloads for 1.7.0-rc.2
 
 
 filename | sha256 hash
 -------- | -----------
-[fission-core-1.6.0-minikube.yaml](https://github.com/fission/fission/releases/download/1.6.0/fission-core-1.6.0-minikube.yaml) | `a9786ff2b73f2c3b67337278707c7fb2d55bcee40524dc38bc9e3ec48410fa8b`
+[fission-all-1.7.0-rc.2-openshift.yaml](https://github.com/fission/fission/releases/download/1.7.0-rc.2/fission-all-1.7.0-rc.2-openshift.yaml) | `67c0ffcb0e156b87237cd3a712a1435f09365701f4c516d5e9a83cb3f1c73041`
-[fission-all-1.6.0.yaml](https://github.com/fission/fission/releases/download/1.6.0/fission-all-1.6.0.yaml) | `ef1c69a69d565a07f82e52936e256f7515eae1bc2b36d111ec43a2e5b6f07755`
+[fission-core-1.7.0-rc.2.yaml](https://github.com/fission/fission/releases/download/1.7.0-rc.2/fission-core-1.7.0-rc.2.yaml) | `79e620b73b6a6b9f3f0290aa5b8b6daf7f77e938e34e839fc8b84cee00351b49`
-[fission-core-1.6.0.yaml](https://github.com/fission/fission/releases/download/1.6.0/fission-core-1.6.0.yaml) | `7640294c0ab7a3192f147c1f17a3949c2f2c76ec9be70509f7a13ad7083318b6`
+[fission-core-1.7.0-rc.2-openshift.yaml](https://github.com/fission/fission/releases/download/1.7.0-rc.2/fission-core-1.7.0-rc.2-openshift.yaml) | `a180331a308500cc6bc936b6c6f262bef69e6d75d5d0a1a54e5912845822ff51`
-[fission-all-1.6.0-minikube.yaml](https://github.com/fission/fission/releases/download/1.6.0/fission-all-1.6.0-minikube.yaml) | `eb07b2ead721b98d4f26ffefcd03497125e8d017d82f1eaf9c0c2541d7c80288`
+[fission-all-1.7.0-rc.2-minikube.yaml](https://github.com/fission/fission/releases/download/1.7.0-rc.2/fission-all-1.7.0-rc.2-minikube.yaml) | `7c43a8acc3125e4a25444cedda15e045c90d911689cb18042ce7c4a0f7b6cc73`
-[fission-core-1.6.0.tgz](https://github.com/fission/fission/releases/download/1.6.0/fission-core-1.6.0.tgz) | `1dfbbc8dfa2f28bb498d3729a4bb608772d32b9644011f5e87d0dd79baa07c23`
+[fission-core-1.7.0-rc.2-minikube.yaml](https://github.com/fission/fission/releases/download/1.7.0-rc.2/fission-core-1.7.0-rc.2-minikube.yaml) | `9c052808104ae833e7d2ed23403f2c84a595f525ef7b9e53dee6b70a810cf032`
-[fission-all-1.6.0.tgz](https://github.com/fission/fission/releases/download/1.6.0/fission-all-1.6.0.tgz) | `83094e61b10a8bbc6ef7697b0f00f9f52f3c2cfb1590fcb7890eb1327d1b3c94`
+[fission-all-1.7.0-rc.2.yaml](https://github.com/fission/fission/releases/download/1.7.0-rc.2/fission-all-1.7.0-rc.2.yaml) | `594ebff290c13ad590a0fa44bbbd45a5b5e0ee6fac853d197e7bc756773c5089`
-[fission-cli-osx](https://github.com/fission/fission/releases/download/1.6.0/fission-cli-osx) | `0f15869c30988667e04780e01ad2ab3c5a51578c6dbbe7a54be43496d1973be5`
+[fission-core-1.7.0-rc.2.tgz](https://github.com/fission/fission/releases/download/1.7.0-rc.2/fission-core-1.7.0-rc.2.tgz) | `8ba194e2bf9cf5ea95b851093675268d1bcb713bdf2f9f8ed0fae26fe7731718`
-[fission-cli-windows.exe](https://github.com/fission/fission/releases/download/1.6.0/fission-cli-windows.exe) | `56f09af9cb26c8f4cf9cd27d06865f2f4b4138ea19fe13e9a0e8d17a2a2d405e`
+[fission-all-1.7.0-rc.2.tgz](https://github.com/fission/fission/releases/download/1.7.0-rc.2/fission-all-1.7.0-rc.2.tgz) | `76359916272b408ee21e18a74e230d5f344bea5d95a050fc931547760a8b52b1`
-[fission-cli-linux](https://github.com/fission/fission/releases/download/1.6.0/fission-cli-linux) | `90b1c14b584d52452089af351d5f615730fb5d5dd626884ec9b2808904b12df8`
+[fission-cli-osx](https://github.com/fission/fission/releases/download/1.7.0-rc.2/fission-cli-osx) | `0b79ba3bd1d74e3fdac12f966753c5268df3f32f7fdab3d11431fdfbd8d3e5d2`
+[fission-cli-windows.exe](https://github.com/fission/fission/releases/download/1.7.0-rc.2/fission-cli-windows.exe) | `9f94e3ad7248b7f5883e78a8a7c0fb41ec7467680be445582e7e5fb1f6a62a50`
+[fission-cli-linux](https://github.com/fission/fission/releases/download/1.7.0-rc.2/fission-cli-linux) | `0f37a1164df007c540e36dead66226fc9919ad5182aa479baabc46da2e9ee72f`
 
 # Change Log
 
-## [1.6.0](https://github.com/fission/fission/tree/1.6.0) (2019-10-10)
+## [1.7.0-rc.2](https://github.com/fission/fission/tree/1.7.0-rc.2) (2019-11-27)
-[Full Changelog](https://github.com/fission/fission/compare/1.5.0...1.6.0)
+[Full Changelog](https://github.com/fission/fission/compare/v1.7.0-rc.2...1.7.0-rc.2)
+
+## [v1.7.0-rc.2](https://github.com/fission/fission/tree/v1.7.0-rc.2) (2019-11-27)
+[Full Changelog](https://github.com/fission/fission/compare/1.7.0-rc.1...v1.7.0-rc.2)
 
 **Merged pull requests:**
 
+- Push extra tag to fit go module semver tag format [\#1444](https://github.com/fission/fission/pull/1444) ([life1347](https://github.com/life1347))
+- Adopt existing orphan kubernetes resources when executor starts up [\#1443](https://github.com/fission/fission/pull/1443) ([life1347](https://github.com/life1347))
+- Revert "Try to fix flaky canary test" [\#1442](https://github.com/fission/fission/pull/1442) ([life1347](https://github.com/life1347))
+- Try to fix flaky canary test [\#1441](https://github.com/fission/fission/pull/1441) ([life1347](https://github.com/life1347))
+- Fix router tries to update ingress when createIngress is false [\#1440](https://github.com/fission/fission/pull/1440) ([life1347](https://github.com/life1347))
+- Fix poolmanager sets 0 timeout for function specialization [\#1439](https://github.com/fission/fission/pull/1439) ([life1347](https://github.com/life1347))
+- Add huge response body test [\#1437](https://github.com/fission/fission/pull/1437) ([life1347](https://github.com/life1347))
+- Return error when specialization failed [\#1436](https://github.com/fission/fission/pull/1436) ([life1347](https://github.com/life1347))
+- Fix poolmanager terminates running function pod periodically [\#1435](https://github.com/fission/fission/pull/1435) ([life1347](https://github.com/life1347))
+- Allow to tap multiple function services at one time [\#1434](https://github.com/fission/fission/pull/1434) ([life1347](https://github.com/life1347))
+- Collect function metrics after finishing request [\#1433](https://github.com/fission/fission/pull/1433) ([life1347](https://github.com/life1347))
+- Fix poolmanager crashes when failed to list environment [\#1432](https://github.com/fission/fission/pull/1432) ([life1347](https://github.com/life1347))
+- Ability to pull builder image from private registry [\#1431](https://github.com/fission/fission/pull/1431) ([life1347](https://github.com/life1347))
+- Add checksum and insecure flag for user to skip checksum generation [\#1430](https://github.com/fission/fission/pull/1430) ([life1347](https://github.com/life1347))
+- Support to set imagePullSecret when creating environment [\#1429](https://github.com/fission/fission/pull/1429) ([life1347](https://github.com/life1347))
+- Fix no kubeobjs get created if fn created before env creation [\#1428](https://github.com/fission/fission/pull/1428) ([life1347](https://github.com/life1347))
+- Fix verbosity flag not found in subcommand [\#1425](https://github.com/fission/fission/pull/1425) ([life1347](https://github.com/life1347))
+- Improve compatibility with Openshift [\#1424](https://github.com/fission/fission/pull/1424) ([life1347](https://github.com/life1347))
+- Fix truncated body returned from router [\#1420](https://github.com/fission/fission/pull/1420) ([life1347](https://github.com/life1347))
+- Fission 1.7.0-rc.1 [\#1419](https://github.com/fission/fission/pull/1419) ([life1347](https://github.com/life1347))
+
+## [1.7.0-rc.1](https://github.com/fission/fission/tree/1.7.0-rc.1) (2019-11-18)
+[Full Changelog](https://github.com/fission/fission/compare/v1.7.0-rc.1...1.7.0-rc.1)
+
+## [v1.7.0-rc.1](https://github.com/fission/fission/tree/v1.7.0-rc.1) (2019-11-18)
+[Full Changelog](https://github.com/fission/fission/compare/1.6.0...v1.7.0-rc.1)
+
+**Merged pull requests:**
+
+- Update release-builder go and helm version [\#1418](https://github.com/fission/fission/pull/1418) ([life1347](https://github.com/life1347))
+- Fix failed to find init\_tools.sh [\#1417](https://github.com/fission/fission/pull/1417) ([life1347](https://github.com/life1347))
+- Support semantic version tags [\#1416](https://github.com/fission/fission/pull/1416) ([life1347](https://github.com/life1347))
+- Show warning when referencing nonexistent resources in spec [\#1415](https://github.com/fission/fission/pull/1415) ([life1347](https://github.com/life1347))
+- Add resource info to output message when saving spec file [\#1414](https://github.com/fission/fission/pull/1414) ([life1347](https://github.com/life1347))
+- Always embed given URL in the archive [\#1413](https://github.com/fission/fission/pull/1413) ([life1347](https://github.com/life1347))
+- Fix wrong command and flag usage description [\#1412](https://github.com/fission/fission/pull/1412) ([life1347](https://github.com/life1347))
+- Add --spec to package command [\#1411](https://github.com/fission/fission/pull/1411) ([life1347](https://github.com/life1347))
+- Drop unreleased features \(record & replay\) [\#1406](https://github.com/fission/fission/pull/1406) ([life1347](https://github.com/life1347))
+- Build error formatting on fission spec apply --wait [\#1403](https://github.com/fission/fission/pull/1403) ([life1347](https://github.com/life1347))
+- Refactor controller client package [\#1402](https://github.com/fission/fission/pull/1402) ([life1347](https://github.com/life1347))
+- Fix fn test failed to query logs from log database [\#1401](https://github.com/fission/fission/pull/1401) ([life1347](https://github.com/life1347))
+- Skip trace for router healthz endpoint [\#1400](https://github.com/fission/fission/pull/1400) ([life1347](https://github.com/life1347))
+- Set jaeger collector endpoint as an environment variable [\#1399](https://github.com/fission/fission/pull/1399) ([life1347](https://github.com/life1347))
+- Replace deprecated serviceAccount with serviceAccountName [\#1398](https://github.com/fission/fission/pull/1398) ([life1347](https://github.com/life1347))
+- Fix helm pre-upgrade check failure problem [\#1397](https://github.com/fission/fission/pull/1397) ([life1347](https://github.com/life1347))
+- Prettify console output message [\#1396](https://github.com/fission/fission/pull/1396) ([life1347](https://github.com/life1347))
+- fix typo [\#1395](https://github.com/fission/fission/pull/1395) ([jjmengze](https://github.com/jjmengze))
+- Reorder command flags and add missing flags [\#1394](https://github.com/fission/fission/pull/1394) ([life1347](https://github.com/life1347))
+- Fix CLI exits with status 0 when error occurs [\#1393](https://github.com/fission/fission/pull/1393) ([life1347](https://github.com/life1347))
+- Poolmanager wait for function pod till specialization timeout [\#1392](https://github.com/fission/fission/pull/1392) ([life1347](https://github.com/life1347))
+- Replace flag text with const [\#1391](https://github.com/fission/fission/pull/1391) ([life1347](https://github.com/life1347))
+- Update prometheus version and disable it from the default installation [\#1389](https://github.com/fission/fission/pull/1389) ([life1347](https://github.com/life1347))
+- Fix helm shows "Not a table" issue when install Fission [\#1387](https://github.com/fission/fission/pull/1387) ([life1347](https://github.com/life1347))
+- Fix githook not aborting push if error occurs [\#1386](https://github.com/fission/fission/pull/1386) ([life1347](https://github.com/life1347))
+- Migrate from urfave/cli to cobra [\#1385](https://github.com/fission/fission/pull/1385) ([life1347](https://github.com/life1347))
+- Update maintainer info [\#1383](https://github.com/fission/fission/pull/1383) ([life1347](https://github.com/life1347))
+- Update Makefile and add git pre-push hook [\#1382](https://github.com/fission/fission/pull/1382) ([life1347](https://github.com/life1347))
+- Update staticcheck version and fix all warnings [\#1381](https://github.com/fission/fission/pull/1381) ([life1347](https://github.com/life1347))
+- Make CLI functions return error instead of fatal out [\#1379](https://github.com/fission/fission/pull/1379) ([life1347](https://github.com/life1347))
+- Refactor record command [\#1378](https://github.com/fission/fission/pull/1378) ([life1347](https://github.com/life1347))
+- Fix reverse proxy shows 404 not found when Istio enabled [\#1377](https://github.com/fission/fission/pull/1377) ([life1347](https://github.com/life1347))
+- Refactor time trigger command [\#1376](https://github.com/fission/fission/pull/1376) ([life1347](https://github.com/life1347))
+- Refactor mqtrigger command [\#1375](https://github.com/fission/fission/pull/1375) ([life1347](https://github.com/life1347))
+- added example for builder podspec [\#1374](https://github.com/fission/fission/pull/1374) ([viveksinghggits](https://github.com/viveksinghggits))
+- Refactor function command [\#1372](https://github.com/fission/fission/pull/1372) ([life1347](https://github.com/life1347))
+- Allow to set API type for tensorflow serving environment [\#1371](https://github.com/fission/fission/pull/1371) ([life1347](https://github.com/life1347))
+- Refactor canary config command [\#1370](https://github.com/fission/fission/pull/1370) ([life1347](https://github.com/life1347))
+- PodSpec support in environment builder [\#1369](https://github.com/fission/fission/pull/1369) ([viveksinghggits](https://github.com/viveksinghggits))
+- Fix utility function uses the wrong flag text to get value [\#1368](https://github.com/fission/fission/pull/1368) ([life1347](https://github.com/life1347))
+- Refactor HTTP trigger command [\#1367](https://github.com/fission/fission/pull/1367) ([life1347](https://github.com/life1347))
+- Update READEME link and add back the basic usage [\#1366](https://github.com/fission/fission/pull/1366) ([life1347](https://github.com/life1347))
+- Refactor kubewatch command [\#1365](https://github.com/fission/fission/pull/1365) ([life1347](https://github.com/life1347))
+- Fix accidentally removed timestamp when listing package [\#1364](https://github.com/fission/fission/pull/1364) ([life1347](https://github.com/life1347))
+- Move route creation to function [\#1362](https://github.com/fission/fission/pull/1362) ([life1347](https://github.com/life1347))
+- Remove UID from CLI output [\#1361](https://github.com/fission/fission/pull/1361) ([life1347](https://github.com/life1347))
+- Allow using URL as archive source when creating functions [\#1360](https://github.com/fission/fission/pull/1360) ([life1347](https://github.com/life1347))
+- Refactor plugin & version subcommands [\#1359](https://github.com/fission/fission/pull/1359) ([life1347](https://github.com/life1347))
+- Provide secrets and configmaps while updating the functions [\#1358](https://github.com/fission/fission/pull/1358) ([viveksinghggits](https://github.com/viveksinghggits))
+- Update fission architecture doc [\#1356](https://github.com/fission/fission/pull/1356) ([life1347](https://github.com/life1347))
+- calling the function that handles kafka messages, asynchronously [\#1355](https://github.com/fission/fission/pull/1355) ([viveksinghggits](https://github.com/viveksinghggits))
+- Fix release script tags wrong image name & sed problem on Linux [\#1353](https://github.com/fission/fission/pull/1353) ([life1347](https://github.com/life1347))
+- Update CHANGELOG for 1.6.0 [\#1352](https://github.com/fission/fission/pull/1352) ([life1347](https://github.com/life1347))
+- Replace `AlwaysSample` with `ProbabilitySampler` in router \(\#1215\) [\#1348](https://github.com/fission/fission/pull/1348) ([ccamel](https://github.com/ccamel))
+- Refactor package CLI command [\#1345](https://github.com/fission/fission/pull/1345) ([life1347](https://github.com/life1347))
+
+## [1.6.0](https://github.com/fission/fission/tree/1.6.0) (2019-10-10)
+[Full Changelog](https://github.com/fission/fission/compare/v1.6.0...1.6.0)
+
+## [v1.6.0](https://github.com/fission/fission/tree/v1.6.0) (2019-10-10)
+[Full Changelog](https://github.com/fission/fission/compare/1.5.0...v1.6.0)
+
+**Merged pull requests:**
+
+- Fission 1.6.0 [\#1351](https://github.com/fission/fission/pull/1351) ([life1347](https://github.com/life1347))
 - Move statefulset to apps/v1 [\#1350](https://github.com/fission/fission/pull/1350) ([life1347](https://github.com/life1347))
 - Fix newdeploy failed to find serviceEntry in cache [\#1349](https://github.com/fission/fission/pull/1349) ([life1347](https://github.com/life1347))
 - Support encoded path in router [\#1347](https://github.com/fission/fission/pull/1347) ([life1347](https://github.com/life1347))
-- Support custom private go vendor [\#1346](https://github.com/fission/fission/pull/1346) ([ti](https://github.com/ti))
+- Support custom private go vendor [\#1346](https://github.com/fission/fission/pull/1346) ([life1347](https://github.com/life1347))
 - Fix the namespace mismatch problem when deploying with a single YAML file [\#1344](https://github.com/fission/fission/pull/1344) ([life1347](https://github.com/life1347))
 - Move charts executor service to the right place [\#1343](https://github.com/fission/fission/pull/1343) ([life1347](https://github.com/life1347))
 - Allow to deploy router as DaemonSet [\#1342](https://github.com/fission/fission/pull/1342) ([life1347](https://github.com/life1347))
-- add package filter feature [\#1341](https://github.com/fission/fission/pull/1341) ([MengZn](https://github.com/MengZn))
+- add package filter feature [\#1341](https://github.com/fission/fission/pull/1341) ([jjmengze](https://github.com/jjmengze))
 - Bump jvm builder JDK version [\#1340](https://github.com/fission/fission/pull/1340) ([life1347](https://github.com/life1347))
 - Fix executor doesn't apply user-configured container spec correctly [\#1339](https://github.com/fission/fission/pull/1339) ([life1347](https://github.com/life1347))
 - Allow to add annotations to router service in helm chart [\#1338](https://github.com/fission/fission/pull/1338) ([prabhu43](https://github.com/prabhu43))
@@ -155,6 +254,7 @@ filename | sha256 hash
 - V1.2.1 [\#1178](https://github.com/fission/fission/pull/1178) ([vishal-biyani](https://github.com/vishal-biyani))
 - Skaffold for Fission [\#1172](https://github.com/fission/fission/pull/1172) ([vishal-biyani](https://github.com/vishal-biyani))
 - Add affinity support [\#1170](https://github.com/fission/fission/pull/1170) ([laurence-hudson-mindfoundry](https://github.com/laurence-hudson-mindfoundry))
+- Using templated imagePullPolicy for containers in deployment.yaml [\#1137](https://github.com/fission/fission/pull/1137) ([msshroff](https://github.com/msshroff))
 - Refactor test framework [\#1128](https://github.com/fission/fission/pull/1128) ([darkgerm](https://github.com/darkgerm))
 - Pod specs [\#1106](https://github.com/fission/fission/pull/1106) ([vishal-biyani](https://github.com/vishal-biyani))
 - Allow non-toplevel modules in python environment [\#1042](https://github.com/fission/fission/pull/1042) ([soamvasani](https://github.com/soamvasani))
@@ -181,13 +281,13 @@ filename | sha256 hash
 - Fix TravisCI go environment version to avoid go bugs [\#1154](https://github.com/fission/fission/pull/1154) ([life1347](https://github.com/life1347))
 - \#1132 nodejs environment, increase body size [\#1149](https://github.com/fission/fission/pull/1149) ([JannikZed](https://github.com/JannikZed))
 - Added php builder to release script fixes \#1140 [\#1145](https://github.com/fission/fission/pull/1145) ([vishal-biyani](https://github.com/vishal-biyani))
-- Using templated imagePullPolicy for containers in deployment.yaml [\#1137](https://github.com/fission/fission/pull/1137) ([msshroff](https://github.com/msshroff))
 - Migrate from glide to official dependencies management tool: Go Module [\#1136](https://github.com/fission/fission/pull/1136) ([life1347](https://github.com/life1347))
 - Fix misleading log when setup portforward [\#1134](https://github.com/fission/fission/pull/1134) ([life1347](https://github.com/life1347))
 - V1.1.0 [\#1129](https://github.com/fission/fission/pull/1129) ([vishal-biyani](https://github.com/vishal-biyani))
 - support KUBECONFIG with multiple kube config files [\#1126](https://github.com/fission/fission/pull/1126) ([grounded042](https://github.com/grounded042))
 - Function update after change in env [\#1116](https://github.com/fission/fission/pull/1116) ([vishal-biyani](https://github.com/vishal-biyani))
 - Add configurable timeout to fission function test [\#1091](https://github.com/fission/fission/pull/1091) ([erwinvaneyk](https://github.com/erwinvaneyk))
+- Add links to examples for each Fission environment [\#1090](https://github.com/fission/fission/pull/1090) ([erwinvaneyk](https://github.com/erwinvaneyk))
 
 ## [1.1.0](https://github.com/fission/fission/tree/1.1.0) (2019-03-25)
 [Full Changelog](https://github.com/fission/fission/compare/1.0.0...1.1.0)
@@ -205,7 +305,6 @@ filename | sha256 hash
 - Added support for Ruby v2 Specialization [\#1101](https://github.com/fission/fission/pull/1101) ([brendanstennett](https://github.com/brendanstennett))
 - V1.0.0 [\#1100](https://github.com/fission/fission/pull/1100) ([vishal-biyani](https://github.com/vishal-biyani))
 - Adding annotations for prometheus scraping to fission-core [\#1098](https://github.com/fission/fission/pull/1098) ([vishal-biyani](https://github.com/vishal-biyani))
-- Add links to examples for each Fission environment [\#1090](https://github.com/fission/fission/pull/1090) ([erwinvaneyk](https://github.com/erwinvaneyk))
 - Switch from fluentd to fluentbit for log forwarding [\#1086](https://github.com/fission/fission/pull/1086) ([soamvasani](https://github.com/soamvasani))
 - Added draft proposal for CI/CD [\#1084](https://github.com/fission/fission/pull/1084) ([vishal-biyani](https://github.com/vishal-biyani))
 - \[Kafka MQT\] Add warning about Kafka version [\#1083](https://github.com/fission/fission/pull/1083) ([bhavin192](https://github.com/bhavin192))
@@ -470,7 +569,6 @@ filename | sha256 hash
 - Changes needed for release 0.7.1 [\#622](https://github.com/fission/fission/pull/622) ([smruthi2187](https://github.com/smruthi2187))
 - Add default value to cli flag [\#619](https://github.com/fission/fission/pull/619) ([life1347](https://github.com/life1347))
 - Remove port forward in tests for router, controller and nats pods  [\#611](https://github.com/fission/fission/pull/611) ([smruthi2187](https://github.com/smruthi2187))
-- updates to changelog. [\#598](https://github.com/fission/fission/pull/598) ([smruthi2187](https://github.com/smruthi2187))
 - meaningful error message when fetch request is received for a package when build is not successful. [\#661](https://github.com/fission/fission/pull/661) ([smruthi2187](https://github.com/smruthi2187))
 - Delete deployment with proper delete propagation policy [\#630](https://github.com/fission/fission/pull/630) ([life1347](https://github.com/life1347))
 - Fix buildmgr SEGFAULT when it failed to update package [\#626](https://github.com/fission/fission/pull/626) ([life1347](https://github.com/life1347))
@@ -485,6 +583,7 @@ filename | sha256 hash
 
 - Prevent releasing idle connections because transport is shared. [\#609](https://github.com/fission/fission/pull/609) ([smruthi2187](https://github.com/smruthi2187))
 - Fix components crash before crds creation [\#602](https://github.com/fission/fission/pull/602) ([life1347](https://github.com/life1347))
+- updates to changelog. [\#598](https://github.com/fission/fission/pull/598) ([smruthi2187](https://github.com/smruthi2187))
 - changes needed for release 0.7.0 [\#597](https://github.com/fission/fission/pull/597) ([smruthi2187](https://github.com/smruthi2187))
 - `fission X create --spec` flags for env and trigger create commands [\#607](https://github.com/fission/fission/pull/607) ([soamvasani](https://github.com/soamvasani))
 - Updating releasing guideliness with a few more details. [\#599](https://github.com/fission/fission/pull/599) ([smruthi2187](https://github.com/smruthi2187))

charts/README.md

@@ -43,10 +43,10 @@ Parameter | Description | Default
 `routerServiceType` | Type of Fission Router service to use. For minikube, set this to NodePort, elsewhere use LoadBalancer or ClusterIP. | `LoadBalancer`
 `repository` | Image base repository | `index.docker.io`
 `image` | Fission image repository | `fission/fission-bundle`
-`imageTag` | Fission image tag | `1.7.0-rc.1`
+`imageTag` | Fission image tag | `1.7.0`
 `pullPolicy` | Image pull policy | `IfNotPresent`
 `fetcherImage` | Fission fetcher repository | `fission/fetcher`
-`fetcherImageTag` | Fission fetcher image tag | `1.7.0-rc.1`
+`fetcherImageTag` | Fission fetcher image tag | `1.7.0`
 `controllerPort` | Fission Controller service port | `31313`
 `routerPort` | Fission Router service port | ` 31314`
 `functionNamespace` | Namespace in which to run fission functions (this is different from the release namespace) | `fission-function`
@@ -66,6 +66,7 @@ Parameter | Description | Default
 `prometheus.serviceEndpoint` | If prometheus.enabled is false, please assign the prometheus service URL that is accessible by components. | `nil`
 `canaryDeployment.enabled` | Set to true if you need canary deployment feature | `true` in `fission-all`, `false` in `fission-core`
 `extraCoreComponentPodConfig` | Extend the container specs for the core fission pods. Can be used to add things like affinty/tolerations/nodeSelectors/etc. | None
+`executor.adoptExistingResources` | If true, executor will try to adopt existing resources created by the old executor instance. | `false`
 `router.deployAsDaemonSet` | Deploy router as DaemonSet instead of Deployment | `false`
 `router.svcAddressMaxRetries` | Max retries times for router to retry on a certain service URL returns from cache/executor | `5`
 `router.svcAddressUpdateTimeout` | The length of update lock expiry time for router to get a service URL returns from executor | `30`

charts/fission-all/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: v1
 name: fission-all
-version: 1.7.0-rc.1
+version: 1.7.0
 description: Fission is a fast serverless framework for Kubernetes.
 keywords:
 - fission
@@ -12,4 +12,4 @@ maintainers:
   - name: Ta Ching Chen
     email: hello@tachingchen.com
 engine: gotpl
-appVersion: 1.7.0-rc.1
+appVersion: 1.7.0

charts/fission-all/requirements.lock

@@ -2,5 +2,5 @@ dependencies:
 - name: prometheus
   repository: https://kubernetes-charts.storage.googleapis.com
   version: 9.3.1
-digest: sha256:33395db6ac17f57dab0e60a47a3e029c59aa4a58fb1695c73b8ea6f0754eaf0c
+digest: sha256:89a63a5f01c08032106a94c59fc7a5929e449458bf5fd06d86c8b048ac628bf4
-generated: 2019-11-09T02:56:08.384381+08:00
+generated: "2019-11-18T10:16:20.334338911Z"

charts/fission-all/templates/deployment.yaml

@@ -215,6 +215,8 @@ spec:
           value: "{{ .Values.pullPolicy }}"
         - name: RUNTIME_IMAGE_PULL_POLICY
           value: "{{ .Values.pullPolicy }}"
+        - name: ADOPT_EXISTING_RESOURCES
+          value: {{ .Values.executor.adoptExistingResources | default false | quote }}
         - name: ENABLE_ISTIO
           value: "{{ .Values.enableIstio }}"
         - name: TRACE_JAEGER_COLLECTOR_ENDPOINT
@@ -534,10 +536,8 @@ spec:
         ]
         ports:
         - containerPort: 4222
-          hostPort: 4222
           protocol: TCP
         - containerPort: 4223
-          hostPort: 4223
           protocol: TCP
         readinessProbe:
           httpGet:

charts/fission-all/templates/fluentbit.yaml

@@ -45,6 +45,10 @@ spec:
             - name: container-log
               mountPath: /var/log/
               readOnly: false
+{{- if .Values.logger.enableSecurityContext }}
+          securityContext:
+            privileged: true
+{{- end }}
       containers:
         - name: logger
 {{ if .Values.repository }}
@@ -68,6 +72,10 @@ spec:
             - name: docker-log
               mountPath: /var/lib/docker/containers
               readOnly: true
+{{- if .Values.logger.enableSecurityContext }}
+          securityContext:
+            privileged: true
+{{- end }}
         - name: fluentbit
 {{- if .Values.repository }}
           image: "{{ .Values.logger.fluentdImageRepository }}/{{ .Values.logger.fluentdImage }}:{{ .Values.logger.fluentdImageTag }}"
@@ -96,6 +104,10 @@ spec:
                   key: password
             - name: LOG_PATH
               value: /var/log/fission/*.log
+{{- if .Values.logger.enableSecurityContext }}
+          securityContext:
+            privileged: true
+{{- end }}
           volumeMounts:
             - name: container-log
               mountPath: /var/log/

charts/fission-all/values.yaml

@@ -20,13 +20,13 @@ image: fission/fission-bundle
 pullPolicy: IfNotPresent
 
 ## Fission image version
-imageTag: 1.7.0-rc.1
+imageTag: 1.7.0
 
 ## Fission fetcher repository
 fetcherImage: fission/fetcher
 
 ## Fission fetcher image version
-fetcherImageTag: 1.7.0-rc.1
+fetcherImageTag: 1.7.0
 
 ## Port at which Fission controller service should be exposed
 controllerPort: 31313
@@ -57,6 +57,16 @@ logger:
   fluentdImageRepository: index.docker.io
   fluentdImage: fluent/fluent-bit
   fluentdImageTag: 1.0.4
+  
+  ## Fluent-bit writes/reads it’s own sqlite database to record a history of tracked
+  ## files and a state of offsets, this is very useful to resume a state if the ser-
+  ## vice is restarted. For Kubernetes environment with constraints like OpenShift,
+  ## the containers are limited to write hostPath volume. Hence, we have to enable
+  ## security context and set privileged to true.
+  enableSecurityContext: false
+
+executor:
+  adoptExistingResources: false
 
 ## Router config
 router:

charts/fission-core/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: v1
 name: fission-core
-version: 1.7.0-rc.1
+version: 1.7.0
 description: Fission is a fast serverless framework for Kubernetes.
 keywords:
 - fission
@@ -12,4 +12,4 @@ maintainers:
   - name: Ta Ching Chen
     email: hello@tachingchen.com
 engine: gotpl
-appVersion: 1.7.0-rc.1
+appVersion: 1.7.0

charts/fission-core/requirements.lock

@@ -2,5 +2,5 @@ dependencies:
 - name: prometheus
   repository: https://kubernetes-charts.storage.googleapis.com
   version: 9.3.1
-digest: sha256:33395db6ac17f57dab0e60a47a3e029c59aa4a58fb1695c73b8ea6f0754eaf0c
+digest: sha256:89a63a5f01c08032106a94c59fc7a5929e449458bf5fd06d86c8b048ac628bf4
-generated: 2019-11-09T02:56:58.282692+08:00
+generated: "2019-11-18T10:16:25.945208326Z"

charts/fission-core/templates/deployment.yaml

@@ -217,6 +217,8 @@ spec:
           value: "{{ .Values.traceCollectorEndpoint }}"
         - name: TRACING_SAMPLING_RATE
           value: {{ .Values.traceSamplingRate | default "0.5" | quote }}
+        - name: ADOPT_EXISTING_RESOURCES
+          value: {{ .Values.executor.adoptExistingResources | default false | quote }}
         - name: ENABLE_ISTIO
           value: "{{ .Values.enableIstio }}"
         - name: FETCHER_MINCPU

charts/fission-core/values.yaml

@@ -17,7 +17,7 @@ repository: index.docker.io
 image: fission/fission-bundle
 
 ## Fission image version
-imageTag: 1.7.0-rc.1
+imageTag: 1.7.0
 
 ## Image pull policy
 pullPolicy: IfNotPresent
@@ -26,7 +26,7 @@ pullPolicy: IfNotPresent
 fetcherImage: fission/fetcher
 
 ## Fission fetcher image version
-fetcherImageTag: 1.7.0-rc.1
+fetcherImageTag: 1.7.0
 
 ## Port at which Fission controller service should be exposed
 controllerPort: 31313
@@ -45,6 +45,9 @@ builderNamespace: fission-builder
 ## Enable istio integration
 enableIstio: false
 
+executor:
+  adoptExistingResources: false
+
 ## Router config
 router:
   deployAsDaemonSet: false

cmd/fission-cli/app/app.go

@@ -37,7 +37,7 @@ func App() *cobra.Command {
 		Use:  "fission",
 		Long: usage,
 		//SilenceUsage: true,
-		PreRunE: wrapper.Wrapper(
+		PersistentPreRunE: wrapper.Wrapper(
 			func(input cli.Input) error {
 				console.Verbosity = input.Int(flagkey.Verbosity)
 				return nil
@@ -53,7 +53,7 @@ func App() *cobra.Command {
 	})
 
 	wrapper.SetFlags(rootCmd, flag.FlagSet{
-		Optional: []flag.Flag{flag.GlobalServer, flag.GlobalVerbosity},
+		Global: []flag.Flag{flag.GlobalServer, flag.GlobalVerbosity},
 	})
 
 	groups := helptemplate.CommandGroups{}
@@ -64,7 +64,9 @@ func App() *cobra.Command {
 	groups = append(groups, helptemplate.CreateCmdGroup("Other Commands", support.Commands(), version.Commands()))
 	groups.Add(rootCmd)
 
-	helptemplate.ActsAsRootCommand(rootCmd, nil, groups...)
+	flagExposer := helptemplate.ActsAsRootCommand(rootCmd, nil, groups...)
+	// show global options in usage
+	flagExposer.ExposeFlags(rootCmd, flagkey.Server, flagkey.Verbosity)
 
 	return rootCmd
 }

hack/Releasing.txt

@@ -75,8 +75,7 @@ a. Switch to fission/fission-charts repo and run `index.sh` in that repo.  (Don'
 
 b. You should have index.yaml and fission-core-*, fission-all-* charts as change, add, commit and push it to repo.
 
-
+9. Go to Github Web UI --> Releases, here you will notice a pre-release for the version x.y.z you just created.
-8. Go to Github Web UI --> Releases, here you will notice a pre-release for the version x.y.z you just created. 
 
 Edit it and add links to:
 
@@ -88,7 +87,7 @@ Before you save the release - UNCHECK the "This is a pre-release" checkbox. This
 Updating Fission Docs (https://github.com/fission/docs.fission.io)
 ------------------------------
 
-9. Documentation Update
+10. Documentation Update
 
 a. Merge documentation PRs that are peer reviewed and get latest master locally.
 

hack/release-build.sh

@@ -198,8 +198,8 @@ build_charts() {
     do
     # https://github.com/kubernetes/helm/issues/1732
     helm init --client-only
-	helm package -u $c/
+	  helm package -u $c/
-	mv *.tgz $BUILDDIR/charts/
+	  mv *.tgz $BUILDDIR/charts/
     done
     popd
 }
@@ -226,6 +226,8 @@ build_yamls() {
         helm template ${c} -n ${releaseName} --namespace fission --set analytics=false,analyticsNonHelmInstall=true,serviceType=NodePort,routerServiceType=NodePort > ${c}-${version}-minikube.yaml
         # for environments that support LoadBalancer
         helm template ${c} -n ${releaseName} --namespace fission --set analytics=false,analyticsNonHelmInstall=true > ${c}-${version}.yaml
+        # for OpenShift
+        helm template ${c} -n ${releaseName} --namespace fission --set analytics=false,analyticsNonHelmInstall=true,logger.enableSecurityContext=true,prometheus.enabled=false > ${c}-${version}-openshift.yaml
 
         # copy yaml files to build directory
         mv *.yaml ${BUILDDIR}/yamls/

hack/release.sh

@@ -14,8 +14,8 @@ check_branch() {
     curr_branch=$(git rev-parse --abbrev-ref HEAD)
     if [ $curr_branch != "release-${version}" ]
     then
-	echo "Not on release-${version} branch."
+        echo "Not on release-${version} branch."
-	exit 1
+        exit 1
     fi
 }
 
@@ -23,8 +23,8 @@ check_branch() {
 check_clean() {    
     if ! git diff-index --quiet HEAD --
     then
-	echo "Unclean tree"
+        echo "Unclean tree"
-	exit 1
+        exit 1
     fi
 }
 
@@ -150,13 +150,22 @@ push_all() {
 
 tag_and_release() {
     local version=$1
-    local gittag="v${version}"
+    local gittag=$version
+    local prefix="v"
+    local gopkgtag=${version/#/${prefix}}
+
+    if [[ ${version} == v* ]]; # if version starts with "v", don't append prefix.
+    then
+        gopkgtag=${version}
+    fi
 
     # tag the release
     git tag $gittag
-    
+    git tag -a $gopkgtag -m "Fission $gopkgtag"
+
     # push tag
     git push origin $gittag
+    git push origin $gopkgtag
 
     # create gh release
     gothub release \
@@ -170,7 +179,7 @@ tag_and_release() {
 
 attach_github_release_cli() {
     local version=$1
-    local gittag="v${version}"
+    local gittag=$version
     # cli
     echo "Uploading osx cli"
     gothub upload \
@@ -195,21 +204,21 @@ attach_github_release_cli() {
 	   --replace \
 	   --user fission \
 	   --repo fission \
-	   --tag $gittag \
+	   --tag  $gittag \
 	   --name fission-cli-windows.exe \
 	   --file $BUILDDIR/cli/windows/fission-cli-windows.exe
 }
 
 attach_github_release_charts() {
     local version=$1
-    local gittag="v${version}"
+    local gittag=$version
 
     # helm charts
     gothub upload \
 	   --replace \
 	   --user fission \
 	   --repo fission \
-	   --tag $gittag \
+	   --tag  $gittag \
 	   --name fission-all-$version.tgz \
 	   --file $BUILDDIR/charts/fission-all-$version.tgz
 
@@ -217,7 +226,7 @@ attach_github_release_charts() {
 	   --replace \
 	   --user fission \
 	   --repo fission \
-	   --tag $gittag \
+	   --tag  $gittag \
 	   --name fission-core-$version.tgz \
 	   --file $BUILDDIR/charts/fission-core-$version.tgz
 
@@ -225,7 +234,7 @@ attach_github_release_charts() {
 
 attach_github_release_yamls() {
     local version=$1
-    local gittag="v${version}"
+    local gittag=$version
 
     for c in fission-all fission-core
     do
@@ -245,6 +254,14 @@ attach_github_release_yamls() {
            --tag $gittag \
            --name ${c}-${version}.yaml \
            --file $BUILDDIR/yamls/${c}-${version}.yaml
+
+        gothub upload \
+           --replace \
+           --user fission \
+           --repo fission \
+           --tag $gittag \
+           --name ${c}-${version}-openshift.yaml \
+           --file $BUILDDIR/yamls/${c}-${version}-openshift.yaml
     done
 }
 

pkg/apis/fission.io/v1/const.go

@@ -18,13 +18,12 @@ package v1
 
 const (
 	EXECUTOR_INSTANCEID_LABEL string = "executorInstanceId"
-	POOLMGR_INSTANCEID_LABEL  string = "poolmgrInstanceId"
 	DEFAULT_FUNCTION_TIMEOUT  int    = 60
 )
 
 const (
-	//LastUpdateTimestamp env variable is used for updating configmaps and secrets in pods
+	// ResourceVersionCount env variable is used for updating configmaps and secrets in pods
-	LastUpdateTimestamp string = "LASTUPDATE_TIMESTAMP"
+	ResourceVersionCount string = "RESOURCE_VERSION_COUNT"
 )
 
 const (
@@ -54,8 +53,8 @@ const (
 )
 
 const (
-	ExecutorTypePoolmgr   = "poolmgr"
+	ExecutorTypePoolmgr   ExecutorType = "poolmgr"
-	ExecutorTypeNewdeploy = "newdeploy"
+	ExecutorTypeNewdeploy ExecutorType = "newdeploy"
 )
 
 const (

pkg/apis/fission.io/v1/types.go

@@ -381,7 +381,6 @@ type (
 		TargetCPUPercent int
 
 		// This is the timeout setting for executor to wait for pod specialization.
-		// Currently, only newdeploy utilizes this value.
 		SpecializationTimeout int
 	}
 
@@ -529,6 +528,10 @@ type (
 		// or unarchived file should be placed, which is then used by specialize handler.
 		// (This is mainly for the JVM environment because .jar is one kind of zip archive.)
 		KeepArchive bool `json:"keeparchive"`
+
+		// ImagePullSecret is the secret for Kubernetes to pull an image from a
+		// private registry.
+		ImagePullSecret string `json:"imagepullsecret"`
 	}
 
 	AllowedFunctionsPerContainer string

pkg/buildermgr/envwatcher.go

@@ -495,7 +495,6 @@ func (envw *environmentWatcher) createBuilderDeployment(env *fv1.Environment, ns
 	}
 
 	finalPodSpec, err := util.MergePodSpec(&podSpec, env.Spec.Builder.PodSpec)
-
 	if err != nil {
 		return nil, err
 	}
@@ -516,7 +515,7 @@ func (envw *environmentWatcher) createBuilderDeployment(env *fv1.Environment, ns
 					Labels:      sel,
 					Annotations: podAnnotations,
 				},
-				Spec: *finalPodSpec,
+				Spec: *(util.ApplyImagePullSecret(env.Spec.ImagePullSecret, *finalPodSpec)),
 			},
 		},
 	}

pkg/controller/client/function.go

@@ -19,10 +19,15 @@ package client
 import (
 	"encoding/json"
 	"fmt"
+	"io"
+	"net/http"
+	"net/url"
 
+	"github.com/pkg/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
 	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
+	"github.com/fission/fission/pkg/fission-cli/console"
 )
 
 func (c *Client) FunctionCreate(f *fv1.Function) (*metav1.ObjectMeta, error) {
@@ -152,3 +157,29 @@ func (c *Client) FunctionList(functionNamespace string) ([]fv1.Function, error)
 
 	return funcs, nil
 }
+
+func (c *Client) FunctionPodLogs(m *metav1.ObjectMeta) (io.ReadCloser, int, error) {
+	relativeUrl := fmt.Sprintf("functions/%v", m.Name)
+	relativeUrl += fmt.Sprintf("?namespace=%v", m.Namespace)
+
+	queryURL, err := url.Parse(c.Url)
+	if err != nil {
+		return nil, 0, errors.Wrapf(err, "error parsing the base URL '%v'", c.Url)
+	}
+	queryURL.Path = fmt.Sprintf("/proxy/logs/%s", m.Name)
+
+	console.Verbose(2, fmt.Sprintf("Try to get pod logs from controller '%v'", queryURL.String()))
+
+	req, err := http.NewRequest(http.MethodPost, queryURL.String(), nil)
+	if err != nil {
+		return nil, 0, errors.Wrap(err, "error creating logs request")
+	}
+
+	httpClient := http.Client{}
+	resp, err := httpClient.Do(req)
+	if err != nil {
+		return nil, 0, errors.Wrap(err, "error executing get logs request")
+	}
+
+	return resp.Body, resp.StatusCode, nil
+}

pkg/controller/functionApi.go

@@ -18,12 +18,15 @@ package controller
 
 import (
 	"encoding/json"
+	"fmt"
 	"io"
 	"io/ioutil"
 	"net/http"
 	"net/http/httputil"
 	"net/url"
 	"sort"
+	"strconv"
+	"strings"
 
 	"github.com/emicklei/go-restful"
 	restfulspec "github.com/emicklei/go-restful-openapi"
@@ -33,10 +36,12 @@ import (
 	"go.uber.org/zap"
 	apiv1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	restclient "k8s.io/client-go/rest"
+	"k8s.io/apimachinery/pkg/labels"
+	"k8s.io/client-go/kubernetes"
 
 	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
 	ferror "github.com/fission/fission/pkg/error"
+	"github.com/fission/fission/pkg/types"
 )
 
 func RegisterFunctionRoute(ws *restful.WebService) {
@@ -280,27 +285,35 @@ func (a *API) FunctionLogsApiPost(w http.ResponseWriter, r *http.Request) {
 func (a *API) FunctionPodLogs(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	fnName := vars["function"]
-	ns := vars["namespace"]
+
+	ns := a.extractQueryParamFromRequest(r, "namespace")
+	podNs := "fission-function"
 
 	if len(ns) == 0 {
-		ns = "fission-function"
+		ns = metav1.NamespaceDefault
+	} else if ns != metav1.NamespaceDefault {
+		// If the function namespace is "default", executor
+		// will create function pods under "fission-function".
+		// Otherwise, the function pod will be created under
+		// the same namespace of function.
+		podNs = ns
 	}
 
 	f, err := a.fissionClient.Functions(ns).Get(fnName)
-	if err != nil {
-		a.respondWithError(w, err)
-		return
-	}
-	envName := f.Spec.Environment.Name
-
 	if err != nil {
 		a.respondWithError(w, err)
 		return
 	}
 
 	// Get function Pods first
-	selector := "functionName=" + fnName
+	selector := map[string]string{
-	podList, err := a.kubernetesClient.CoreV1().Pods(ns).List(metav1.ListOptions{LabelSelector: selector})
+		types.FUNCTION_UID:          string(f.Metadata.UID),
+		types.ENVIRONMENT_NAME:      f.Spec.Environment.Name,
+		types.ENVIRONMENT_NAMESPACE: f.Spec.Environment.Namespace,
+	}
+	podList, err := a.kubernetesClient.CoreV1().Pods(podNs).List(metav1.ListOptions{
+		LabelSelector: labels.Set(selector).AsSelector().String(),
+	})
 	if err != nil {
 		a.respondWithError(w, err)
 		return
@@ -309,30 +322,47 @@ func (a *API) FunctionPodLogs(w http.ResponseWriter, r *http.Request) {
 	// Get the logs for last Pod executed
 	pods := podList.Items
 	sort.Slice(pods, func(i, j int) bool {
-		itime := pods[i].ObjectMeta.CreationTimestamp.Time
+		rv1, _ := strconv.ParseInt(pods[i].ObjectMeta.ResourceVersion, 10, 32)
-		jtime := pods[j].ObjectMeta.CreationTimestamp.Time
+		rv2, _ := strconv.ParseInt(pods[j].ObjectMeta.ResourceVersion, 10, 32)
-		return itime.After(jtime)
+		return rv1 > rv2
 	})
 
-	podLogOpts := apiv1.PodLogOptions{Container: envName} // Only the env container, not fetcher
+	if len(pods) <= 0 {
-	var podLogsReq *restclient.Request
-	if len(pods) > 0 {
-		podLogsReq = a.kubernetesClient.CoreV1().Pods(ns).GetLogs(pods[0].ObjectMeta.Name, &podLogOpts)
-	} else {
 		a.respondWithError(w, errors.New("no active pods found"))
 		return
 	}
 
-	podLogs, err := podLogsReq.Stream()
+	// get the pod with highest resource version
+	err = getContainerLog(a.kubernetesClient, w, f, &pods[0])
 	if err != nil {
-		a.respondWithError(w, err)
+		a.respondWithError(w, errors.Wrapf(err, "error getting container logs"))
-		return
-	}
-	defer podLogs.Close()
-
-	_, err = io.Copy(w, podLogs)
-	if err != nil {
-		a.respondWithError(w, err)
 		return
 	}
 }
+
+func getContainerLog(kubernetesClient *kubernetes.Clientset, w http.ResponseWriter, fn *fv1.Function, pod *apiv1.Pod) error {
+	seq := strings.Repeat("=", 35)
+
+	for _, container := range pod.Spec.Containers {
+		podLogOpts := apiv1.PodLogOptions{Container: container.Name} // Only the env container, not fetcher
+		podLogsReq := kubernetesClient.CoreV1().Pods(pod.Namespace).GetLogs(pod.ObjectMeta.Name, &podLogOpts)
+
+		podLogs, err := podLogsReq.Stream()
+		if err != nil {
+			return errors.Wrapf(err, "error streaming pod log")
+		}
+
+		msg := fmt.Sprintf("\n%v\nFunction: %v\nEnvironment: %v\nNamespace: %v\nPod: %v\nContainer: %v\nNode: %v\n%v\n", seq,
+			fn.Metadata.Name, fn.Spec.Environment.Name, pod.Namespace, pod.Name, container.Name, pod.Spec.NodeName, seq)
+		w.Write([]byte(msg))
+
+		_, err = io.Copy(w, podLogs)
+		if err != nil {
+			return errors.Wrapf(err, "error copying pod log")
+		}
+
+		podLogs.Close()
+	}
+
+	return nil
+}

pkg/executor/api.go

@@ -17,7 +17,6 @@ limitations under the License.
 package executor
 
 import (
-	"context"
 	"encoding/json"
 	"fmt"
 	"io/ioutil"
@@ -25,6 +24,8 @@ import (
 	"strings"
 
 	"github.com/gorilla/mux"
+	"github.com/hashicorp/go-multierror"
+	"github.com/pkg/errors"
 	"go.opencensus.io/plugin/ochttp"
 	"go.uber.org/zap"
 	k8serrors "k8s.io/apimachinery/pkg/api/errors"
@@ -32,6 +33,7 @@ import (
 
 	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
 	ferror "github.com/fission/fission/pkg/error"
+	"github.com/fission/fission/pkg/executor/client"
 )
 
 func (executor *Executor) getServiceForFunctionApi(w http.ResponseWriter, r *http.Request) {
@@ -88,9 +90,15 @@ func (executor *Executor) getServiceForFunction(fn *fv1.Function) (string, error
 		zap.String("function_name", fn.Metadata.Name),
 		zap.String("function_namespace", fn.Metadata.Namespace))
 
-	fsvc, err := executor.fsCache.GetByFunction(&fn.Metadata)
+	t := fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType
+	et, exists := executor.executorTypes[t]
+	if !exists {
+		return "", errors.Errorf("Unknown executor type '%v'", t)
+	}
+
+	fsvc, err := et.GetFuncSvcFromCache(fn)
 	if err == nil {
-		if executor.isValidAddress(fsvc) {
+		if et.IsValid(fsvc) {
 			// Cached, return svc address
 			return fsvc.Address, nil
 		} else {
@@ -98,7 +106,7 @@ func (executor *Executor) getServiceForFunction(fn *fv1.Function) (string, error
 				zap.String("function_name", fn.Metadata.Name),
 				zap.String("function_namespace", fn.Metadata.Namespace),
 				zap.String("address", fsvc.Address))
-			executor.fsCache.DeleteEntry(fsvc)
+			et.DeleteFuncSvcFromCache(fsvc)
 		}
 	}
 
@@ -115,25 +123,57 @@ func (executor *Executor) getServiceForFunction(fn *fv1.Function) (string, error
 }
 
 // find funcSvc and update its atime
+// TODO: Deprecated tapService
 func (executor *Executor) tapService(w http.ResponseWriter, r *http.Request) {
+	// only for upgrade compatibility
+	w.WriteHeader(http.StatusOK)
+}
+
+// find funcSvc and update its atime
+func (executor *Executor) tapServices(w http.ResponseWriter, r *http.Request) {
 	body, err := ioutil.ReadAll(r.Body)
 	if err != nil {
 		executor.logger.Error("failed to read tap service request", zap.Error(err))
 		http.Error(w, "Failed to read request", http.StatusInternalServerError)
 		return
 	}
-	svcName := string(body)
-	svcHost := strings.TrimPrefix(svcName, "http://")
 
-	err = executor.fsCache.TouchByAddress(svcHost)
+	tapSvcReqs := []client.TapServiceRequest{}
+	err = json.Unmarshal(body, &tapSvcReqs)
 	if err != nil {
-		executor.logger.Error("error tapping function service",
+		executor.logger.Error("failed to decode tap service request",
 			zap.Error(err),
-			zap.String("service", svcName),
+			zap.String("request-payload", string(body)))
-			zap.String("host", svcHost))
+		http.Error(w, "Failed to decode tap service request", http.StatusBadRequest)
+		return
+	}
+
+	errs := &multierror.Error{}
+	for _, req := range tapSvcReqs {
+		svcHost := strings.TrimPrefix(req.ServiceUrl, "http://")
+
+		et, exists := executor.executorTypes[req.FnExecutorType]
+		if !exists {
+			errs = multierror.Append(errs,
+				errors.Errorf("error tapping service due to unknown executor type '%v' found",
+					req.FnExecutorType))
+			continue
+		}
+
+		err = et.TapService(svcHost)
+		if err != nil {
+			errs = multierror.Append(errs,
+				errors.Wrapf(err, "'%v' failed to tap function '%v' in '%v' with service url '%v'",
+					req.FnMetadata.Name, req.FnMetadata.Namespace, req.ServiceUrl, req.FnExecutorType))
+		}
+	}
+
+	if errs.ErrorOrNil() != nil {
+		executor.logger.Error("error tapping function service", zap.Error(errs))
 		http.Error(w, "Not found", http.StatusNotFound)
 		return
 	}
+
 	w.WriteHeader(http.StatusOK)
 }
 
@@ -144,21 +184,14 @@ func (executor *Executor) healthHandler(w http.ResponseWriter, r *http.Request)
 func (executor *Executor) GetHandler() http.Handler {
 	r := mux.NewRouter()
 	r.HandleFunc("/v2/getServiceForFunction", executor.getServiceForFunctionApi).Methods("POST")
-	r.HandleFunc("/v2/tapService", executor.tapService).Methods("POST")
+	r.HandleFunc("/v2/tapService", executor.tapService).Methods("POST") // for backward compatibility
+	r.HandleFunc("/v2/tapServices", executor.tapServices).Methods("POST")
 	r.HandleFunc("/healthz", executor.healthHandler).Methods("GET")
 	return r
 }
 
 func (executor *Executor) Serve(port int) {
-	executor.logger.Info("starting executor", zap.Int("port", port))
+	executor.logger.Info("starting executor API", zap.Int("port", port))
-
-	ctx, cancel := context.WithCancel(context.Background())
-	defer cancel()
-
-	executor.ndm.Run(ctx)
-	executor.gpm.Run(ctx)
-	executor.cms.Run(ctx)
-
 	address := fmt.Sprintf(":%v", port)
 	err := http.ListenAndServe(address, &ochttp.Handler{
 		Handler: executor.GetHandler(),

pkg/executor/client/client.go

@@ -32,23 +32,31 @@ import (
 	"golang.org/x/net/context/ctxhttp"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
+	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
 	ferror "github.com/fission/fission/pkg/error"
 )
 
-type Client struct {
+type (
-	logger      *zap.Logger
+	Client struct {
-	executorUrl string
+		logger      *zap.Logger
-	tappedByUrl map[string]bool
+		executorUrl string
-	requestChan chan string
+		tappedByUrl map[string]TapServiceRequest
-	httpClient  *http.Client
+		requestChan chan TapServiceRequest
-}
+		httpClient  *http.Client
+	}
+	TapServiceRequest struct {
+		FnMetadata     metav1.ObjectMeta
+		FnExecutorType fv1.ExecutorType
+		ServiceUrl     string
+	}
+)
 
 func MakeClient(logger *zap.Logger, executorUrl string) *Client {
 	c := &Client{
 		logger:      logger.Named("executor_client"),
 		executorUrl: strings.TrimSuffix(executorUrl, "/"),
-		tappedByUrl: make(map[string]bool),
+		tappedByUrl: make(map[string]TapServiceRequest),
-		requestChan: make(chan string),
+		requestChan: make(chan TapServiceRequest, 100),
 		httpClient: &http.Client{
 			Transport: &ochttp.Transport{},
 		},
@@ -87,40 +95,64 @@ func (c *Client) service() {
 	ticker := time.NewTicker(time.Second * 5)
 	for {
 		select {
-		case serviceUrl := <-c.requestChan:
+		case svcReq := <-c.requestChan:
-			c.tappedByUrl[serviceUrl] = true
+			c.tappedByUrl[svcReq.ServiceUrl] = svcReq
 		case <-ticker.C:
-			urls := c.tappedByUrl
+			if len(c.tappedByUrl) == 0 {
-			c.tappedByUrl = make(map[string]bool)
+				continue
-			if len(urls) > 0 {
-				go func() {
-					for u := range urls {
-						err := c._tapService(u)
-						if err != nil {
-							c.logger.Error("error tapping function service address", zap.Error(err), zap.String("address", u))
-						}
-					}
-					c.logger.Debug("tapped services in batch", zap.Int("service_count", len(urls)))
-				}()
 			}
+
+			urls := c.tappedByUrl
+			c.tappedByUrl = make(map[string]TapServiceRequest)
+
+			go func() {
+				svcReqs := []TapServiceRequest{}
+				for _, req := range urls {
+					svcReqs = append(svcReqs, req)
+				}
+				c.logger.Debug("tapped services in batch", zap.Int("service_count", len(urls)))
+
+				err := c._tapService(svcReqs)
+				if err != nil {
+					c.logger.Error("error tapping function service address", zap.Error(err))
+				}
+			}()
 		}
 	}
 }
 
-func (c *Client) TapService(serviceUrl *url.URL) {
+func (c *Client) TapService(fnMeta metav1.ObjectMeta, executorType fv1.ExecutorType, serviceUrl *url.URL) {
-	c.requestChan <- serviceUrl.String()
+	c.requestChan <- TapServiceRequest{
+		FnMetadata: metav1.ObjectMeta{
+			Name:            fnMeta.Name,
+			Namespace:       fnMeta.Namespace,
+			ResourceVersion: fnMeta.ResourceVersion,
+			UID:             fnMeta.UID,
+		},
+		FnExecutorType: executorType,
+		// service url is for executor to know which
+		// pod/service is currently used to serve user function.
+		ServiceUrl: serviceUrl.String(),
+	}
 }
 
-func (c *Client) _tapService(serviceUrlStr string) error {
+func (c *Client) _tapService(tapSvcReqs []TapServiceRequest) error {
-	executorUrl := c.executorUrl + "/v2/tapService"
+	executorUrl := c.executorUrl + "/v2/tapServices"
 
-	resp, err := http.Post(executorUrl, "application/octet-stream", bytes.NewReader([]byte(serviceUrlStr)))
+	body, err := json.Marshal(tapSvcReqs)
+	if err != nil {
+		return err
+	}
+
+	resp, err := http.Post(executorUrl, "application/json", bytes.NewReader(body))
 	if err != nil {
 		return err
 	}
 	defer resp.Body.Close()
+
 	if resp.StatusCode != 200 {
 		return ferror.MakeErrorFromHTTP(resp)
 	}
+
 	return nil
 }

pkg/executor/cms/cmscontroller.go

@@ -20,6 +20,7 @@ import (
 	"context"
 	"time"
 
+	"github.com/pkg/errors"
 	"go.uber.org/zap"
 	apiv1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -29,8 +30,7 @@ import (
 
 	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
 	"github.com/fission/fission/pkg/crd"
-	nd "github.com/fission/fission/pkg/executor/newdeploy"
+	"github.com/fission/fission/pkg/executor/executortype"
-	gpm "github.com/fission/fission/pkg/executor/poolmgr"
 )
 
 type (
@@ -46,10 +46,10 @@ type (
 
 //MakeConfigSecretController makes a controller for configmaps and secrets which changes related functions
 func MakeConfigSecretController(logger *zap.Logger, fissionClient *crd.FissionClient,
-	kubernetesClient *kubernetes.Clientset, ndm *nd.NewDeploy, gpm *gpm.GenericPoolManager) *ConfigSecretController {
+	kubernetesClient *kubernetes.Clientset, types map[fv1.ExecutorType]executortype.ExecutorType) *ConfigSecretController {
 	logger.Debug("Creating ConfigMap & Secret Controller")
-	_, cmcontroller := initConfigmapController(logger, fissionClient, kubernetesClient, ndm, gpm)
+	_, cmcontroller := initConfigmapController(logger, fissionClient, kubernetesClient, types)
-	_, scontroller := initSecretController(logger, fissionClient, kubernetesClient, ndm, gpm)
+	_, scontroller := initSecretController(logger, fissionClient, kubernetesClient, types)
 	cmsController := &ConfigSecretController{
 		logger:              logger,
 		configmapController: cmcontroller,
@@ -66,7 +66,7 @@ func (csController *ConfigSecretController) Run(ctx context.Context) {
 }
 
 func initConfigmapController(logger *zap.Logger, fissionClient *crd.FissionClient,
-	kubernetesClient *kubernetes.Clientset, ndm *nd.NewDeploy, gpm *gpm.GenericPoolManager) (cache.Store, cache.Controller) {
+	kubernetesClient *kubernetes.Clientset, types map[fv1.ExecutorType]executortype.ExecutorType) (cache.Store, cache.Controller) {
 	resyncPeriod := 30 * time.Second
 	listWatch := cache.NewListWatchFromClient(kubernetesClient.CoreV1().RESTClient(), "configmaps", metav1.NamespaceAll, fields.Everything())
 	store, controller := cache.NewInformer(listWatch, &apiv1.ConfigMap{}, resyncPeriod, cache.ResourceEventHandlerFuncs{
@@ -81,12 +81,11 @@ func initConfigmapController(logger *zap.Logger, fissionClient *crd.FissionClien
 						zap.String("configmap_name", newCm.ObjectMeta.Name),
 						zap.String("configmap_namespace", newCm.ObjectMeta.Namespace))
 				}
-
 				funcs, err := getConfigmapRelatedFuncs(logger, &newCm.ObjectMeta, fissionClient)
 				if err != nil {
-					logger.Error("Failed to get functions related to secret", zap.String("secret_name", newCm.ObjectMeta.Name), zap.String("secret_namespace", newCm.ObjectMeta.Namespace))
+					logger.Error("Failed to get functions related to configmap", zap.String("configmap_name", newCm.ObjectMeta.Name), zap.String("configmap_namespace", newCm.ObjectMeta.Namespace))
 				}
-				recyclePods(logger, funcs, ndm, gpm)
+				refreshPods(logger, funcs, types)
 			}
 		},
 	})
@@ -112,7 +111,7 @@ func getConfigmapRelatedFuncs(logger *zap.Logger, m *metav1.ObjectMeta, fissionC
 }
 
 func initSecretController(logger *zap.Logger, fissionClient *crd.FissionClient,
-	kubernetesClient *kubernetes.Clientset, ndm *nd.NewDeploy, gpm *gpm.GenericPoolManager) (cache.Store, cache.Controller) {
+	kubernetesClient *kubernetes.Clientset, types map[fv1.ExecutorType]executortype.ExecutorType) (cache.Store, cache.Controller) {
 	resyncPeriod := 30 * time.Second
 	listWatch := cache.NewListWatchFromClient(kubernetesClient.CoreV1().RESTClient(), "secrets", metav1.NamespaceAll, fields.Everything())
 	store, controller := cache.NewInformer(listWatch, &apiv1.Secret{}, resyncPeriod, cache.ResourceEventHandlerFuncs{
@@ -128,14 +127,12 @@ func initSecretController(logger *zap.Logger, fissionClient *crd.FissionClient,
 						zap.String("configmap_namespace", newS.ObjectMeta.Namespace))
 
 				}
-
 				funcs, err := getSecretRelatedFuncs(logger, &newS.ObjectMeta, fissionClient)
 				if err != nil {
 					logger.Error("Failed to get functions related to secret", zap.String("secret_name", newS.ObjectMeta.Name), zap.String("secret_namespace", newS.ObjectMeta.Namespace))
 				}
-				recyclePods(logger, funcs, ndm, gpm)
+				refreshPods(logger, funcs, types)
 			}
-
 		},
 	})
 	return store, controller
@@ -160,19 +157,19 @@ func getSecretRelatedFuncs(logger *zap.Logger, m *metav1.ObjectMeta, fissionClie
 	return relatedFunctions, nil
 }
 
-func recyclePods(logger *zap.Logger, funcs []fv1.Function, ndm *nd.NewDeploy, gpm *gpm.GenericPoolManager) {
+func refreshPods(logger *zap.Logger, funcs []fv1.Function, types map[fv1.ExecutorType]executortype.ExecutorType) {
 	for _, f := range funcs {
 		var err error
 
-		switch f.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType {
+		et, exists := types[f.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType]
-		case fv1.ExecutorTypeNewdeploy:
+		if exists {
-			err = ndm.RefreshFuncPods(logger, f)
+			err = et.RefreshFuncPods(logger, f)
-		case fv1.ExecutorTypePoolmgr:
+		} else {
-			err = gpm.RefreshFuncPods(logger, f)
+			err = errors.Errorf("Unknown executor type '%v'", f.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType)
 		}
 
 		if err != nil {
-			logger.Error("Failed to recycle pods for function after configmap changed",
+			logger.Error("Failed to recycle pods for function after configmap/secret changed",
 				zap.Error(err),
 				zap.Any("function", f))
 		}

pkg/executor/executor.go

@@ -20,6 +20,8 @@ import (
 	"context"
 	"fmt"
 	"net/http"
+	"os"
+	"strconv"
 	"strings"
 	"sync"
 	"time"
@@ -32,10 +34,12 @@ import (
 	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
 	"github.com/fission/fission/pkg/crd"
 	"github.com/fission/fission/pkg/executor/cms"
+	"github.com/fission/fission/pkg/executor/executortype"
+	"github.com/fission/fission/pkg/executor/executortype/newdeploy"
+	"github.com/fission/fission/pkg/executor/executortype/poolmgr"
 	"github.com/fission/fission/pkg/executor/fscache"
-	"github.com/fission/fission/pkg/executor/newdeploy"
-	"github.com/fission/fission/pkg/executor/poolmgr"
 	"github.com/fission/fission/pkg/executor/reaper"
+	"github.com/fission/fission/pkg/executor/util"
 	fetcherConfig "github.com/fission/fission/pkg/fetcher/config"
 )
 
@@ -43,12 +47,10 @@ type (
 	Executor struct {
 		logger *zap.Logger
 
-		gpm *poolmgr.GenericPoolManager
+		executorTypes map[fv1.ExecutorType]executortype.ExecutorType
-		ndm *newdeploy.NewDeploy
+		cms           *cms.ConfigSecretController
-		cms *cms.ConfigSecretController
 
 		fissionClient *crd.FissionClient
-		fsCache       *fscache.FunctionServiceCache
 
 		requestChan chan *createFuncServiceRequest
 		fsCreateWg  map[string]*sync.WaitGroup
@@ -64,21 +66,26 @@ type (
 	}
 )
 
-func MakeExecutor(logger *zap.Logger, gpm *poolmgr.GenericPoolManager, ndm *newdeploy.NewDeploy, cms *cms.ConfigSecretController, fissionClient *crd.FissionClient, fsCache *fscache.FunctionServiceCache) *Executor {
+func MakeExecutor(logger *zap.Logger, cms *cms.ConfigSecretController,
+	fissionClient *crd.FissionClient, types map[fv1.ExecutorType]executortype.ExecutorType) (*Executor, error) {
 	executor := &Executor{
 		logger:        logger.Named("executor"),
-		gpm:           gpm,
-		ndm:           ndm,
 		cms:           cms,
 		fissionClient: fissionClient,
-		fsCache:       fsCache,
+		executorTypes: types,
 
 		requestChan: make(chan *createFuncServiceRequest),
 		fsCreateWg:  make(map[string]*sync.WaitGroup),
 	}
+	for _, et := range types {
+		go func(et executortype.ExecutorType) {
+			et.Run(context.Background())
+		}(et)
+	}
+	go cms.Run(context.Background())
 	go executor.serveCreateFuncServices()
 
-	return executor
+	return executor, nil
 }
 
 // All non-cached function service requests go through this goroutine
@@ -113,8 +120,18 @@ func (executor *Executor) serveCreateFuncServices() {
 				// still can serve other subsequent requests.
 
 				buffer := 10 // add some buffer time for specialization
+				specializationTimeout := req.function.Spec.InvokeStrategy.ExecutionStrategy.SpecializationTimeout
+
+				// set minimum specialization timeout to avoid illegal input and
+				// compatibility problem when applying old spec file that doesn't
+				// have specialization timeout field.
+				if specializationTimeout < fv1.DefaultSpecializationTimeOut {
+					specializationTimeout = fv1.DefaultSpecializationTimeOut
+				}
+
 				fnSpecializationTimeoutContext, cancel := context.WithTimeout(context.Background(),
-					time.Duration(req.function.Spec.InvokeStrategy.ExecutionStrategy.SpecializationTimeout+buffer)*time.Second)
+					time.Duration(specializationTimeout+buffer)*time.Second)
+				defer cancel()
 
 				fsvc, err := executor.createServiceForFunction(fnSpecializationTimeoutContext, req.function)
 				req.respChan <- &createFuncServiceResponse{
@@ -122,8 +139,6 @@ func (executor *Executor) serveCreateFuncServices() {
 					err:     err,
 				}
 				delete(executor.fsCreateWg, crd.CacheKey(fnMetadata))
-
-				cancel()
 				wg.Done()
 			}()
 		} else {
@@ -134,7 +149,7 @@ func (executor *Executor) serveCreateFuncServices() {
 				wg.Wait()
 
 				// get the function service from the cache
-				fsvc, err := executor.fsCache.GetByFunction(fnMetadata)
+				fsvc, err := executor.getFunctionServiceFromCache(req.function)
 
 				// fsCache return error when the entry does not exist/expire.
 				// It normally happened if there are multiple requests are
@@ -155,18 +170,13 @@ func (executor *Executor) createServiceForFunction(ctx context.Context, fn *fv1.
 		zap.String("function_name", fn.Metadata.Name),
 		zap.String("function_namespace", fn.Metadata.Namespace))
 
-	executorType := fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType
+	t := fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType
-
+	e, ok := executor.executorTypes[t]
-	var fsvc *fscache.FuncSvc
+	if !ok {
-	var fsvcErr error
+		return nil, errors.Errorf("Unknown executor type '%v'", t)
-
-	switch executorType {
-	case fv1.ExecutorTypeNewdeploy:
-		fsvc, fsvcErr = executor.ndm.GetFuncSvc(ctx, fn)
-	default:
-		fsvc, fsvcErr = executor.gpm.GetFuncSvc(ctx, fn)
 	}
 
+	fsvc, fsvcErr := e.GetFuncSvc(ctx, fn)
 	if fsvcErr != nil {
 		e := "error creating service for function"
 		executor.logger.Error(e,
@@ -174,25 +184,18 @@ func (executor *Executor) createServiceForFunction(ctx context.Context, fn *fv1.
 			zap.String("function_name", fn.Metadata.Name),
 			zap.String("function_namespace", fn.Metadata.Namespace))
 		fsvcErr = errors.Wrap(fsvcErr, fmt.Sprintf("[%s] %s", fn.Metadata.Name, e))
-	} else if fsvc != nil {
-		_, err := executor.fsCache.Add(*fsvc)
-		if err != nil {
-			return nil, err
-		}
 	}
 
-	executor.fsCache.IncreaseColdStarts(fn.Metadata.Name, string(fn.Metadata.UID))
-
 	return fsvc, fsvcErr
 }
 
-// isValidAddress invokes isValidService or isValidPod depending on the type of executor
+func (executor *Executor) getFunctionServiceFromCache(fn *fv1.Function) (*fscache.FuncSvc, error) {
-func (executor *Executor) isValidAddress(fsvc *fscache.FuncSvc) bool {
+	t := fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType
-	if fsvc.Executor == fscache.NEWDEPLOY {
+	e, ok := executor.executorTypes[t]
-		return executor.ndm.IsValid(fsvc)
+	if !ok {
-	} else {
+		return nil, errors.Errorf("Unknown executor type '%v'", t)
-		return executor.gpm.IsValid(fsvc)
 	}
+	return e.GetFuncSvcFromCache(fn)
 }
 
 func serveMetric(logger *zap.Logger) {
@@ -223,26 +226,49 @@ func StartExecutor(logger *zap.Logger, functionNamespace string, envBuilderNames
 	}
 
 	restClient := fissionClient.GetCrdClient()
-	fsCache := fscache.MakeFunctionServiceCache(logger)
+	executorInstanceID := strings.ToLower(uniuri.NewLen(8))
 
-	poolID := strings.ToLower(uniuri.NewLen(8))
+	logger.Info("Starting executor", zap.String("instanceID", executorInstanceID))
-	reaper.CleanupOldExecutorObjects(logger, kubernetesClient, poolID)
-	go reaper.CleanupRoleBindings(logger, kubernetesClient, fissionClient, functionNamespace, envBuilderNamespace, time.Minute*30)
 
 	gpm := poolmgr.MakeGenericPoolManager(
 		logger,
 		fissionClient, kubernetesClient,
-		functionNamespace, fetcherConfig, poolID)
+		functionNamespace, fetcherConfig, executorInstanceID)
 
 	ndm := newdeploy.MakeNewDeploy(
 		logger,
 		fissionClient, kubernetesClient, restClient,
-		functionNamespace, fetcherConfig, poolID)
+		functionNamespace, fetcherConfig, executorInstanceID)
 
-	cms := cms.MakeConfigSecretController(logger, fissionClient, kubernetesClient, ndm, gpm)
+	executorTypes := make(map[fv1.ExecutorType]executortype.ExecutorType)
+	executorTypes[gpm.GetTypeName()] = gpm
+	executorTypes[ndm.GetTypeName()] = ndm
 
-	api := MakeExecutor(logger, gpm, ndm, cms, fissionClient, fsCache)
+	adoptExistingResources, _ := strconv.ParseBool(os.Getenv("ADOPT_EXISTING_RESOURCES"))
 
+	wg := &sync.WaitGroup{}
+	for _, et := range executorTypes {
+		wg.Add(1)
+		go func(et executortype.ExecutorType) {
+			defer wg.Done()
+			if adoptExistingResources {
+				et.AdoptExistingResources()
+			}
+			et.CleanupOldExecutorObjects()
+		}(et)
+	}
+	// set hard timeout for resource adoption
+	// TODO: use context to control the waiting time once kubernetes client supports it.
+	util.WaitTimeout(wg, 30*time.Second)
+
+	cms := cms.MakeConfigSecretController(logger, fissionClient, kubernetesClient, executorTypes)
+
+	api, err := MakeExecutor(logger, cms, fissionClient, executorTypes)
+	if err != nil {
+		return err
+	}
+
+	go reaper.CleanupRoleBindings(logger, kubernetesClient, fissionClient, functionNamespace, envBuilderNamespace, time.Minute*30)
 	go api.Serve(port)
 	go serveMetric(logger)
 

pkg/executor/executortype/executortype.go

@@ -0,0 +1,59 @@
+/*
+Copyright 2019 The Fission Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package executortype
+
+import (
+	"context"
+	"go.uber.org/zap"
+
+	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
+	"github.com/fission/fission/pkg/executor/fscache"
+)
+
+type ExecutorType interface {
+	// Run runs background job.
+	Run(context.Context)
+
+	// GetTypeName returns the name of executor type
+	GetTypeName() fv1.ExecutorType
+
+	// GetFuncSvc specializes function pod(s) and returns a service URL for the function.
+	GetFuncSvc(context.Context, *fv1.Function) (*fscache.FuncSvc, error)
+
+	// GetFuncSvcFromCache retrieves function service from cache.
+	GetFuncSvcFromCache(*fv1.Function) (*fscache.FuncSvc, error)
+
+	// DeleteFuncSvcFromCache deletes function service entry in cache.
+	DeleteFuncSvcFromCache(*fscache.FuncSvc)
+
+	// TapService updates the access time of function service entry to
+	// avoid idle pod reaper recycles pods.
+	TapService(serviceUrl string) error
+
+	// IsValid returns true if a function service is valid. Different executor types
+	// use distinct ways to examine the function service.
+	IsValid(*fscache.FuncSvc) bool
+
+	// RefreshFuncPods refreshes function pods if the secrets/configmaps pods reference to get updated.
+	RefreshFuncPods(*zap.Logger, fv1.Function) error
+
+	// AdoptOrphanResources adopts existing resources created by the deleted executor.
+	AdoptExistingResources()
+
+	// CleanupOldExecutorObjects cleans up resources created by old executor instances
+	CleanupOldExecutorObjects()
+}

pkg/executor/executortype/newdeploy/newdeploy.go

@@ -17,11 +17,12 @@ limitations under the License.
 package newdeploy
 
 import (
-	"errors"
 	"fmt"
+	"strconv"
 	"time"
 
 	multierror "github.com/hashicorp/go-multierror"
+	"github.com/pkg/errors"
 	"go.uber.org/zap"
 	appsv1 "k8s.io/api/apps/v1"
 	asv1 "k8s.io/api/autoscaling/v1"
@@ -30,6 +31,7 @@ import (
 	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/util/intstr"
+	"k8s.io/client-go/kubernetes"
 
 	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
 	"github.com/fission/fission/pkg/executor/util"
@@ -43,32 +45,53 @@ const (
 )
 
 func (deploy *NewDeploy) createOrGetDeployment(fn *fv1.Function, env *fv1.Environment,
-	deployName string, deployLabels map[string]string, deployNamespace string, firstcreate bool) (*appsv1.Deployment, error) {
+	deployName string, deployLabels map[string]string, deployAnnotations map[string]string, deployNamespace string) (*appsv1.Deployment, error) {
 
-	minScale := int32(fn.Spec.InvokeStrategy.ExecutionStrategy.MinScale)
 	specializationTimeout := int(fn.Spec.InvokeStrategy.ExecutionStrategy.SpecializationTimeout)
+	minScale := int32(fn.Spec.InvokeStrategy.ExecutionStrategy.MinScale)
 
-	// If it's not the first time creation and minscale is 0 means that all pods for function were recycled,
+	// Always scale to at least one pod when createOrGetDeployment
-	// in such cases we need set minscale to 1 for router to serve requests.
+	// is called. The idleObjectReaper will scale-in the deployment
-	if !firstcreate && minScale <= 0 {
+	// later if no requests to the function.
+	if minScale <= 0 {
 		minScale = 1
 	}
 
-	waitForDeploy := minScale > 0
+	deployment, err := deploy.getDeploymentSpec(fn, env, &minScale, deployName, deployNamespace, deployLabels, deployAnnotations)
+	if err != nil {
+		return nil, err
+	}
 
 	existingDepl, err := deploy.kubernetesClient.AppsV1().Deployments(deployNamespace).Get(deployName, metav1.GetOptions{})
 	if err == nil {
-		if waitForDeploy {
+		// Try to adopt orphan deployment created by the old executor.
+		if existingDepl.Annotations[fv1.EXECUTOR_INSTANCEID_LABEL] != deploy.instanceID {
+			existingDepl.Annotations = deployment.Annotations
+			existingDepl.Labels = deployment.Labels
+			existingDepl.Spec.Template.Spec.Containers = deployment.Spec.Template.Spec.Containers
+			existingDepl.Spec.Template.Spec.ServiceAccountName = deployment.Spec.Template.Spec.ServiceAccountName
+			existingDepl.Spec.Template.Spec.TerminationGracePeriodSeconds = deployment.Spec.Template.Spec.TerminationGracePeriodSeconds
+			existingDepl, err = deploy.kubernetesClient.AppsV1().Deployments(deployNamespace).Update(existingDepl)
+			if err != nil {
+				deploy.logger.Warn("error adopting deploy", zap.Error(err),
+					zap.String("deploy", deployName), zap.String("ns", deployNamespace))
+				return nil, err
+			}
+			// In this case, we just return without waiting for it for fast bootstraping.
+			return existingDepl, nil
+		}
+
+		if *existingDepl.Spec.Replicas < minScale {
 			err = deploy.scaleDeployment(existingDepl.Namespace, existingDepl.Name, minScale)
 			if err != nil {
 				deploy.logger.Error("error scaling up function deployment", zap.Error(err), zap.String("function", fn.Metadata.Name))
 				return nil, err
 			}
-
-			if existingDepl.Status.AvailableReplicas < minScale {
-				existingDepl, err = deploy.waitForDeploy(existingDepl, minScale, specializationTimeout)
-			}
 		}
+		if existingDepl.Status.AvailableReplicas < minScale {
+			existingDepl, err = deploy.waitForDeploy(existingDepl, minScale, specializationTimeout)
+		}
+
 		return existingDepl, err
 	} else if k8s_err.IsNotFound(err) {
 		err := deploy.setupRBACObjs(deployNamespace, fn)
@@ -76,30 +99,26 @@ func (deploy *NewDeploy) createOrGetDeployment(fn *fv1.Function, env *fv1.Enviro
 			return nil, err
 		}
 
-		deployment, err := deploy.getDeploymentSpec(fn, env, deployName, deployLabels)
-		if err != nil {
-			return nil, err
-		}
-
 		depl, err := deploy.kubernetesClient.AppsV1().Deployments(deployNamespace).Create(deployment)
 		if err != nil {
-			deploy.logger.Error("error while creating function deployment",
+			if k8s_err.IsAlreadyExists(err) {
-				zap.Error(err),
+				depl, err = deploy.kubernetesClient.AppsV1().Deployments(deployNamespace).Get(deployName, metav1.GetOptions{})
-				zap.String("function", fn.Metadata.Name),
+			}
-				zap.String("deployment_name", deployName),
+			if err != nil {
-				zap.String("deployment_namespace", deployNamespace))
+				deploy.logger.Error("error while creating function deployment",
-			return nil, err
+					zap.Error(err),
+					zap.String("function", fn.Metadata.Name),
+					zap.String("deployment_name", deployName),
+					zap.String("deployment_namespace", deployNamespace))
+				return nil, err
+			}
 		}
-
+		if minScale > 0 {
-		if waitForDeploy {
 			depl, err = deploy.waitForDeploy(depl, minScale, specializationTimeout)
 		}
-
 		return depl, err
 	}
-
 	return nil, err
-
 }
 
 func (deploy *NewDeploy) setupRBACObjs(deployNamespace string, fn *fv1.Function) error {
@@ -157,10 +176,13 @@ func (deploy *NewDeploy) deleteDeployment(ns string, name string) error {
 	})
 }
 
-func (deploy *NewDeploy) getDeploymentSpec(fn *fv1.Function, env *fv1.Environment,
+func (deploy *NewDeploy) getDeploymentSpec(fn *fv1.Function, env *fv1.Environment, targetReplicas *int32,
-	deployName string, deployLabels map[string]string) (*appsv1.Deployment, error) {
+	deployName string, deployNamespace string, deployLabels map[string]string, deployAnnotations map[string]string) (*appsv1.Deployment, error) {
 
 	replicas := int32(fn.Spec.InvokeStrategy.ExecutionStrategy.MinScale)
+	if targetReplicas != nil {
+		replicas = *targetReplicas
+	}
 
 	gracePeriodSeconds := int64(6 * 60)
 	if env.Spec.TerminationGracePeriod > 0 {
@@ -171,6 +193,10 @@ func (deploy *NewDeploy) getDeploymentSpec(fn *fv1.Function, env *fv1.Environmen
 	if podAnnotations == nil {
 		podAnnotations = make(map[string]string)
 	}
+
+	// Here, we don't append deployAnnotations to podAnnotations
+	// since newdeploy doesn't manager pod lifecycle directly.
+
 	if deploy.useIstio && env.Spec.AllowAccessToExternalNetwork {
 		podAnnotations["sidecar.istio.io/inject"] = "false"
 	}
@@ -193,6 +219,11 @@ func (deploy *NewDeploy) getDeploymentSpec(fn *fv1.Function, env *fv1.Environmen
 	// rollback, set RevisionHistoryLimit to 0 to disable this feature.
 	revisionHistoryLimit := int32(0)
 
+	rvCount, err := referencedResourcesRVSum(deploy.kubernetesClient, fn.Metadata.Namespace, fn.Spec.Secrets, fn.Spec.ConfigMaps)
+	if err != nil {
+		return nil, err
+	}
+
 	container, err := util.MergeContainer(&apiv1.Container{
 		Name:                   fn.Metadata.Name,
 		Image:                  env.Spec.Runtime.Image,
@@ -210,8 +241,8 @@ func (deploy *NewDeploy) getDeploymentSpec(fn *fv1.Function, env *fv1.Environmen
 		},
 		Env: []apiv1.EnvVar{
 			{
-				Name:  fv1.LastUpdateTimestamp,
+				Name:  fv1.ResourceVersionCount,
-				Value: time.Now().String(),
+				Value: fmt.Sprintf("%v", rvCount),
 			},
 		},
 		// https://istio.io/docs/setup/kubernetes/additional-setup/requirements/
@@ -227,27 +258,32 @@ func (deploy *NewDeploy) getDeploymentSpec(fn *fv1.Function, env *fv1.Environmen
 		return nil, err
 	}
 
+	pod := apiv1.PodTemplateSpec{
+		ObjectMeta: metav1.ObjectMeta{
+			Labels:      deployLabels,
+			Annotations: podAnnotations,
+		},
+		Spec: apiv1.PodSpec{
+			Containers:                    []apiv1.Container{*container},
+			ServiceAccountName:            "fission-fetcher",
+			TerminationGracePeriodSeconds: &gracePeriodSeconds,
+		},
+	}
+
+	pod.Spec = *(util.ApplyImagePullSecret(env.Spec.ImagePullSecret, pod.Spec))
+
 	deployment := &appsv1.Deployment{
 		ObjectMeta: metav1.ObjectMeta{
-			Name:   deployName,
+			Name:        deployName,
-			Labels: deployLabels,
+			Labels:      deployLabels,
+			Annotations: deployAnnotations,
 		},
 		Spec: appsv1.DeploymentSpec{
 			Replicas: &replicas,
 			Selector: &metav1.LabelSelector{
 				MatchLabels: deployLabels,
 			},
-			Template: apiv1.PodTemplateSpec{
+			Template: pod,
-				ObjectMeta: metav1.ObjectMeta{
-					Labels:      deployLabels,
-					Annotations: podAnnotations,
-				},
-				Spec: apiv1.PodSpec{
-					Containers:                    []apiv1.Container{*container},
-					ServiceAccountName:            "fission-fetcher",
-					TerminationGracePeriodSeconds: &gracePeriodSeconds,
-				},
-			},
 			Strategy: appsv1.DeploymentStrategy{
 				Type: appsv1.RollingUpdateDeploymentStrategyType,
 				RollingUpdate: &appsv1.RollingUpdateDeployment{
@@ -315,7 +351,12 @@ func (deploy *NewDeploy) getResources(env *fv1.Environment, fn *fv1.Function) ap
 	return resources
 }
 
-func (deploy *NewDeploy) createOrGetHpa(hpaName string, execStrategy *fv1.ExecutionStrategy, depl *appsv1.Deployment) (*asv1.HorizontalPodAutoscaler, error) {
+func (deploy *NewDeploy) createOrGetHpa(hpaName string, execStrategy *fv1.ExecutionStrategy,
+	depl *appsv1.Deployment, deployLabels map[string]string, deployAnnotations map[string]string) (*asv1.HorizontalPodAutoscaler, error) {
+
+	if depl == nil {
+		return nil, errors.New("failed to create HPA, found empty deployment")
+	}
 
 	minRepl := int32(execStrategy.MinScale)
 	if minRepl == 0 {
@@ -327,42 +368,52 @@ func (deploy *NewDeploy) createOrGetHpa(hpaName string, execStrategy *fv1.Execut
 	}
 	targetCPU := int32(execStrategy.TargetCPUPercent)
 
+	hpa := &asv1.HorizontalPodAutoscaler{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:        hpaName,
+			Labels:      deployLabels,
+			Annotations: deployAnnotations,
+		},
+		Spec: asv1.HorizontalPodAutoscalerSpec{
+			ScaleTargetRef: asv1.CrossVersionObjectReference{
+				Kind:       DeploymentKind,
+				Name:       depl.ObjectMeta.Name,
+				APIVersion: DeploymentVersion,
+			},
+			MinReplicas:                    &minRepl,
+			MaxReplicas:                    maxRepl,
+			TargetCPUUtilizationPercentage: &targetCPU,
+		},
+	}
+
 	existingHpa, err := deploy.kubernetesClient.AutoscalingV1().HorizontalPodAutoscalers(depl.ObjectMeta.Namespace).Get(hpaName, metav1.GetOptions{})
 	if err == nil {
-		return existingHpa, err
+		// to adopt orphan service
-	}
+		if existingHpa.Annotations[fv1.EXECUTOR_INSTANCEID_LABEL] != deploy.instanceID {
-
+			existingHpa.Annotations = hpa.Annotations
-	if depl == nil {
+			existingHpa.Labels = hpa.Labels
-		return nil, errors.New("failed to create HPA, found empty deployment")
+			existingHpa.Spec = hpa.Spec
-	}
+			existingHpa, err = deploy.kubernetesClient.AutoscalingV1().HorizontalPodAutoscalers(depl.ObjectMeta.Namespace).Update(existingHpa)
-
+			if err != nil {
-	if err != nil && k8s_err.IsNotFound(err) {
+				deploy.logger.Warn("error adopting HPA", zap.Error(err),
-		hpa := asv1.HorizontalPodAutoscaler{
+					zap.String("HPA", hpaName), zap.String("ns", depl.ObjectMeta.Namespace))
-			ObjectMeta: metav1.ObjectMeta{
+				return nil, err
-				Name:   hpaName,
+			}
-				Labels: depl.Labels,
-			},
-			Spec: asv1.HorizontalPodAutoscalerSpec{
-				ScaleTargetRef: asv1.CrossVersionObjectReference{
-					Kind:       DeploymentKind,
-					Name:       depl.ObjectMeta.Name,
-					APIVersion: DeploymentVersion,
-				},
-				MinReplicas:                    &minRepl,
-				MaxReplicas:                    maxRepl,
-				TargetCPUUtilizationPercentage: &targetCPU,
-			},
 		}
-
+		return existingHpa, err
-		cHpa, err := deploy.kubernetesClient.AutoscalingV1().HorizontalPodAutoscalers(depl.ObjectMeta.Namespace).Create(&hpa)
+	} else if k8s_err.IsNotFound(err) {
+		cHpa, err := deploy.kubernetesClient.AutoscalingV1().HorizontalPodAutoscalers(depl.ObjectMeta.Namespace).Create(hpa)
 		if err != nil {
-			return nil, err
+			if k8s_err.IsAlreadyExists(err) {
+				cHpa, err = deploy.kubernetesClient.AutoscalingV1().HorizontalPodAutoscalers(depl.ObjectMeta.Namespace).Get(hpaName, metav1.GetOptions{})
+			}
+			if err != nil {
+				return nil, err
+			}
 		}
 		return cHpa, nil
 	}
-
 	return nil, err
-
 }
 
 func (deploy *NewDeploy) getHpa(ns, name string) (*asv1.HorizontalPodAutoscaler, error) {
@@ -378,32 +429,52 @@ func (deploy *NewDeploy) deleteHpa(ns string, name string) error {
 	return deploy.kubernetesClient.AutoscalingV1().HorizontalPodAutoscalers(ns).Delete(name, &metav1.DeleteOptions{})
 }
 
-func (deploy *NewDeploy) createOrGetSvc(deployLabels map[string]string, svcName string, svcNamespace string) (*apiv1.Service, error) {
+func (deploy *NewDeploy) createOrGetSvc(deployLabels map[string]string, deployAnnotations map[string]string, svcName string, svcNamespace string) (*apiv1.Service, error) {
+	service := &apiv1.Service{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:        svcName,
+			Labels:      deployLabels,
+			Annotations: deployAnnotations,
+		},
+		Spec: apiv1.ServiceSpec{
+			Ports: []apiv1.ServicePort{
+				{
+					Name:       "http-env",
+					Port:       int32(80),
+					TargetPort: intstr.FromInt(8888),
+				},
+			},
+			Selector: deployLabels,
+			Type:     apiv1.ServiceTypeClusterIP,
+		},
+	}
+
 	existingSvc, err := deploy.kubernetesClient.CoreV1().Services(svcNamespace).Get(svcName, metav1.GetOptions{})
 	if err == nil {
+		// to adopt orphan service
+		if existingSvc.Annotations[fv1.EXECUTOR_INSTANCEID_LABEL] != deploy.instanceID {
+			existingSvc.Annotations = service.Annotations
+			existingSvc.Labels = service.Labels
+			existingSvc.Spec.Ports = service.Spec.Ports
+			existingSvc.Spec.Selector = service.Spec.Selector
+			existingSvc.Spec.Type = service.Spec.Type
+			existingSvc, err = deploy.kubernetesClient.CoreV1().Services(svcNamespace).Update(existingSvc)
+			if err != nil {
+				deploy.logger.Warn("error adopting service", zap.Error(err),
+					zap.String("service", svcName), zap.String("ns", svcNamespace))
+				return nil, err
+			}
+		}
 		return existingSvc, err
 	} else if k8s_err.IsNotFound(err) {
-		service := &apiv1.Service{
-			ObjectMeta: metav1.ObjectMeta{
-				Name:   svcName,
-				Labels: deployLabels,
-			},
-			Spec: apiv1.ServiceSpec{
-				Ports: []apiv1.ServicePort{
-					{
-						Name:       "http-env",
-						Port:       int32(80),
-						TargetPort: intstr.FromInt(8888),
-					},
-				},
-				Selector: deployLabels,
-				Type:     apiv1.ServiceTypeClusterIP,
-			},
-		}
-
 		svc, err := deploy.kubernetesClient.CoreV1().Services(svcNamespace).Create(service)
 		if err != nil {
-			return nil, err
+			if k8s_err.IsAlreadyExists(err) {
+				svc, err = deploy.kubernetesClient.CoreV1().Services(svcNamespace).Get(svcName, metav1.GetOptions{})
+			}
+			if err != nil {
+				return nil, err
+			}
 		}
 		return svc, nil
 	}
@@ -472,3 +543,57 @@ func (deploy *NewDeploy) cleanupNewdeploy(ns string, name string) error {
 
 	return result.ErrorOrNil()
 }
+
+// referencedResourcesRVSum returns the sum of resource version of all resources the function references to.
+// We used to update timestamp in the deployment environment field in order to trigger a rolling update when
+// the function referenced resources get updated. However, use timestamp means we are not able to avoid tri-
+// ggering a rolling update when executor tries to adopt orphaned deployment due to timestamp changed which
+// is unwanted. In order to let executor adopt deployment without triggering a rolling update, we need an
+// identical way to get a value that can reflect resources changed without affecting by the time.
+// To achieve this goal, the sum of the resource version of all referenced resources is a good fit for our
+// scenario since the sum of the resource version is always the same as long as no resources changed.
+func referencedResourcesRVSum(client *kubernetes.Clientset, namespace string, secrets []fv1.SecretReference, cfgmaps []fv1.ConfigMapReference) (int, error) {
+	rvCount := 0
+
+	if len(secrets) > 0 {
+		list, err := client.CoreV1().Secrets(namespace).List(metav1.ListOptions{})
+		if err != nil {
+			return 0, err
+		}
+
+		objmap := make(map[string]apiv1.Secret)
+		for _, secret := range list.Items {
+			objmap[secret.Namespace+"/"+secret.Name] = secret
+		}
+
+		for _, ref := range secrets {
+			s, ok := objmap[ref.Namespace+"/"+ref.Name]
+			if ok {
+				rv, _ := strconv.ParseInt(s.ResourceVersion, 10, 32)
+				rvCount += int(rv)
+			}
+		}
+	}
+
+	if len(cfgmaps) > 0 {
+		list, err := client.CoreV1().ConfigMaps(namespace).List(metav1.ListOptions{})
+		if err != nil {
+			return 0, err
+		}
+
+		objmap := make(map[string]apiv1.ConfigMap)
+		for _, cfg := range list.Items {
+			objmap[cfg.Namespace+"/"+cfg.Name] = cfg
+		}
+
+		for _, ref := range cfgmaps {
+			s, ok := objmap[ref.Namespace+"/"+ref.Name]
+			if ok {
+				rv, _ := strconv.ParseInt(s.ResourceVersion, 10, 32)
+				rvCount += int(rv)
+			}
+		}
+	}
+
+	return rvCount, nil
+}

pkg/executor/executortype/newdeploy/newdeploymgr.go

@@ -22,11 +22,9 @@ import (
 	"os"
 	"strconv"
 	"strings"
+	"sync"
 	"time"
 
-	"github.com/dchest/uniuri"
-	"github.com/fission/fission/pkg/throttler"
-	"github.com/fission/fission/pkg/utils"
 	multierror "github.com/hashicorp/go-multierror"
 	"github.com/pkg/errors"
 	"go.uber.org/zap"
@@ -43,11 +41,17 @@ import (
 
 	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
 	"github.com/fission/fission/pkg/crd"
+	"github.com/fission/fission/pkg/executor/executortype"
 	"github.com/fission/fission/pkg/executor/fscache"
+	"github.com/fission/fission/pkg/executor/reaper"
 	fetcherConfig "github.com/fission/fission/pkg/fetcher/config"
+	"github.com/fission/fission/pkg/throttler"
 	"github.com/fission/fission/pkg/types"
+	"github.com/fission/fission/pkg/utils"
 )
 
+var _ executortype.ExecutorType = &NewDeploy{}
+
 type (
 	NewDeploy struct {
 		logger *zap.Logger
@@ -83,7 +87,7 @@ func MakeNewDeploy(
 	namespace string,
 	fetcherConfig *fetcherConfig.Config,
 	instanceID string,
-) *NewDeploy {
+) executortype.ExecutorType {
 	enableIstio := false
 	if len(os.Getenv("ENABLE_ISTIO")) > 0 {
 		istio, err := strconv.ParseBool(os.Getenv("ENABLE_ISTIO"))
@@ -131,6 +135,173 @@ func (deploy *NewDeploy) Run(ctx context.Context) {
 	go deploy.idleObjectReaper()
 }
 
+func (deploy *NewDeploy) GetTypeName() fv1.ExecutorType {
+	return fv1.ExecutorTypeNewdeploy
+}
+
+func (deploy *NewDeploy) GetFuncSvc(ctx context.Context, fn *fv1.Function) (*fscache.FuncSvc, error) {
+	// TODO: client-go doesn't support to pass in context.
+	//  Once it supports context, we should change the signature of method.
+	// https://github.com/kubernetes/kubernetes/issues/46503
+	return deploy.createFunction(fn)
+}
+
+func (deploy *NewDeploy) GetFuncSvcFromCache(fn *fv1.Function) (*fscache.FuncSvc, error) {
+	return deploy.fsCache.GetByFunction(&fn.Metadata)
+}
+
+func (deploy *NewDeploy) DeleteFuncSvcFromCache(fsvc *fscache.FuncSvc) {
+	deploy.fsCache.DeleteEntry(fsvc)
+}
+
+func (deploy *NewDeploy) TapService(svcHost string) error {
+	err := deploy.fsCache.TouchByAddress(svcHost)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+// IsValid does a get on the service address to ensure it's a valid service, then
+// scale deployment to 1 replica if there are no available replicas for function.
+// Return true if no error occurs, return false otherwise.
+func (deploy *NewDeploy) IsValid(fsvc *fscache.FuncSvc) bool {
+	service := strings.Split(fsvc.Address, ".")
+	if len(service) == 0 {
+		return false
+	}
+
+	_, err := deploy.kubernetesClient.CoreV1().Services(service[1]).Get(service[0], metav1.GetOptions{})
+	if err != nil {
+		if !k8sErrs.IsNotFound(err) {
+			deploy.logger.Error("error validating function service address", zap.String("function", fsvc.Function.Name), zap.Error(err))
+		}
+		return false
+	}
+
+	deployObj := getDeploymentObj(fsvc.KubernetesObjects)
+	if deployObj == nil {
+		deploy.logger.Error("deployment obj for function does not exist", zap.String("function", fsvc.Function.Name))
+		return false
+	}
+
+	currentDeploy, err := deploy.kubernetesClient.AppsV1().
+		Deployments(deployObj.Namespace).Get(deployObj.Name, metav1.GetOptions{})
+	if err != nil {
+		if !k8sErrs.IsNotFound(err) {
+			deploy.logger.Error("error validating function deployment", zap.Error(err), zap.String("function", fsvc.Function.Name))
+		}
+		return false
+	}
+
+	// return directly when available replicas > 0
+	if currentDeploy.Status.AvailableReplicas > 0 {
+		return true
+	}
+
+	return false
+}
+
+// RefreshFuncPods deleted pods related to the function so that new pods are replenished
+func (deploy *NewDeploy) RefreshFuncPods(logger *zap.Logger, f fv1.Function) error {
+
+	env, err := deploy.fissionClient.Environments(f.Spec.Environment.Namespace).Get(f.Spec.Environment.Name)
+	if err != nil {
+		return err
+	}
+
+	funcLabels := deploy.getDeployLabels(f.Metadata, metav1.ObjectMeta{
+		Name:      f.Spec.Environment.Name,
+		Namespace: f.Spec.Environment.Namespace,
+		UID:       env.Metadata.UID,
+	})
+
+	dep, err := deploy.kubernetesClient.AppsV1().Deployments(metav1.NamespaceAll).List(metav1.ListOptions{
+		LabelSelector: labels.Set(funcLabels).AsSelector().String(),
+	})
+
+	if err != nil {
+		return err
+	}
+
+	// Ideally there should be only one deployment but for now we rely on label/selector to ensure that condition
+	for _, deployment := range dep.Items {
+		rvCount, err := referencedResourcesRVSum(deploy.kubernetesClient, deployment.Namespace, f.Spec.Secrets, f.Spec.ConfigMaps)
+		if err != nil {
+			return err
+		}
+
+		patch := fmt.Sprintf(`{"spec" : {"template": {"spec":{"containers":[{"name": "%s", "env":[{"name": "%s", "value": "%v"}]}]}}}}`,
+			f.Metadata.Name, fv1.ResourceVersionCount, rvCount)
+
+		_, err = deploy.kubernetesClient.AppsV1().Deployments(deployment.ObjectMeta.Namespace).Patch(deployment.ObjectMeta.Name,
+			k8sTypes.StrategicMergePatchType,
+			[]byte(patch))
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (deploy *NewDeploy) AdoptExistingResources() {
+	fnList, err := deploy.fissionClient.Functions(metav1.NamespaceAll).List(metav1.ListOptions{})
+	if err != nil {
+		deploy.logger.Error("error getting function list", zap.Error(err))
+		return
+	}
+
+	wg := &sync.WaitGroup{}
+
+	for i := range fnList.Items {
+		fn := &fnList.Items[i]
+		if fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType == fv1.ExecutorTypeNewdeploy {
+			wg.Add(1)
+			go func() {
+				defer wg.Done()
+
+				_, err = deploy.fnCreate(fn)
+				if err != nil {
+					deploy.logger.Warn("failed to adopt resources for function", zap.Error(err))
+					return
+				}
+				deploy.logger.Info("adopt resources for function", zap.String("function", fn.Metadata.Name))
+			}()
+		}
+	}
+
+	wg.Wait()
+}
+
+func (deploy *NewDeploy) CleanupOldExecutorObjects() {
+	deploy.logger.Info("Newdeploy starts to clean orphaned resources", zap.String("instanceID", deploy.instanceID))
+
+	errs := &multierror.Error{}
+	listOpts := metav1.ListOptions{
+		LabelSelector: labels.Set(map[string]string{types.EXECUTOR_TYPE: string(fv1.ExecutorTypeNewdeploy)}).AsSelector().String(),
+	}
+
+	err := reaper.CleanupHpa(deploy.logger, deploy.kubernetesClient, deploy.instanceID, listOpts)
+	if err != nil {
+		errs = multierror.Append(errs, err)
+	}
+
+	err = reaper.CleanupDeployments(deploy.logger, deploy.kubernetesClient, deploy.instanceID, listOpts)
+	if err != nil {
+		errs = multierror.Append(errs, err)
+	}
+
+	err = reaper.CleanupServices(deploy.logger, deploy.kubernetesClient, deploy.instanceID, listOpts)
+	if err != nil {
+		errs = multierror.Append(errs, err)
+	}
+
+	if errs.ErrorOrNil() != nil {
+		// TODO retry reaper; logged and ignored for now
+		deploy.logger.Error("Failed to cleanup old executor objects", zap.Error(err))
+	}
+}
+
 func (deploy *NewDeploy) initFuncController() (k8sCache.Store, k8sCache.Controller) {
 	resyncPeriod := 30 * time.Second
 	listWatch := k8sCache.NewListWatchFromClient(deploy.crdClient, "functions", metav1.NamespaceAll, fields.Everything())
@@ -142,7 +313,7 @@ func (deploy *NewDeploy) initFuncController() (k8sCache.Store, k8sCache.Controll
 			go func() {
 				fn := obj.(*fv1.Function)
 				deploy.logger.Debug("create deployment for function", zap.Any("fn", fn.Metadata), zap.Any("fnspec", fn.Spec))
-				_, err := deploy.createFunction(fn, true)
+				_, err := deploy.createFunction(fn)
 				if err != nil {
 					deploy.logger.Error("error eager creating function",
 						zap.Error(err),
@@ -224,57 +395,14 @@ func (deploy *NewDeploy) getEnvFunctions(m *metav1.ObjectMeta) []fv1.Function {
 	return relatedFunctions
 }
 
-func (deploy *NewDeploy) GetFuncSvc(ctx context.Context, fn *fv1.Function) (*fscache.FuncSvc, error) {
+func (deploy *NewDeploy) createFunction(fn *fv1.Function) (*fscache.FuncSvc, error) {
-	return deploy.createFunction(fn, false)
-}
-
-// RefreshFuncPods deleted pods related to the function so that new pods are replenished
-func (deploy *NewDeploy) RefreshFuncPods(logger *zap.Logger, f fv1.Function) error {
-
-	env, err := deploy.fissionClient.Environments(f.Spec.Environment.Namespace).Get(f.Spec.Environment.Name)
-	if err != nil {
-		return err
-	}
-
-	funcLabels := deploy.getDeployLabels(f.Metadata, metav1.ObjectMeta{
-		Name:      f.Spec.Environment.Name,
-		Namespace: f.Spec.Environment.Namespace,
-		UID:       env.Metadata.UID,
-	})
-
-	dep, err := deploy.kubernetesClient.AppsV1().Deployments(metav1.NamespaceAll).List(metav1.ListOptions{
-		LabelSelector: labels.Set(funcLabels).AsSelector().String(),
-	})
-
-	if err != nil {
-		return err
-	}
-
-	patch := fmt.Sprintf(`{"spec" : {"template": {"spec":{"containers":[{"name": "%s", "env":[{"name": "%s", "value": "%s"}]}]}}}}`,
-		f.Metadata.Name,
-		fv1.LastUpdateTimestamp,
-		time.Now().String())
-
-	// Ideally there should be only one deployment but for now we rely on label/selector to ensure that condition
-	for _, deployment := range dep.Items {
-		_, err := deploy.kubernetesClient.AppsV1().Deployments(deployment.ObjectMeta.Namespace).Patch(deployment.ObjectMeta.Name,
-			k8sTypes.StrategicMergePatchType,
-			[]byte(patch))
-		if err != nil {
-			return err
-		}
-	}
-	return nil
-}
-
-func (deploy *NewDeploy) createFunction(fn *fv1.Function, firstcreate bool) (*fscache.FuncSvc, error) {
 	if fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType != fv1.ExecutorTypeNewdeploy {
 		return nil, nil
 	}
 
 	fsvcObj, err := deploy.throttler.RunOnce(string(fn.Metadata.UID), func(ableToCreate bool) (interface{}, error) {
 		if ableToCreate {
-			return deploy.fnCreate(fn, firstcreate)
+			return deploy.fnCreate(fn)
 		}
 		return deploy.fsCache.GetByFunctionUID(fn.Metadata.UID)
 	})
@@ -307,7 +435,7 @@ func (deploy *NewDeploy) deleteFunction(fn *fv1.Function) error {
 	return err
 }
 
-func (deploy *NewDeploy) fnCreate(fn *fv1.Function, firstcreate bool) (*fscache.FuncSvc, error) {
+func (deploy *NewDeploy) fnCreate(fn *fv1.Function) (*fscache.FuncSvc, error) {
 	env, err := deploy.fissionClient.
 		Environments(fn.Spec.Environment.Namespace).
 		Get(fn.Spec.Environment.Name)
@@ -316,15 +444,8 @@ func (deploy *NewDeploy) fnCreate(fn *fv1.Function, firstcreate bool) (*fscache.
 	}
 
 	objName := deploy.getObjName(fn)
-	if !firstcreate {
-		// retrieve back the previous obj name for later use.
-		fsvc, err := deploy.fsCache.GetByFunctionUID(fn.Metadata.UID)
-		if err != nil {
-			return nil, errors.Wrap(err, "error getting existed function service cache")
-		}
-		objName = fsvc.Name
-	}
 	deployLabels := deploy.getDeployLabels(fn.Metadata, env.Metadata)
+	deployAnnotations := deploy.getDeployAnnotations(fn.Metadata)
 
 	// to support backward compatibility, if the function was created in default ns, we fall back to creating the
 	// deployment of the function in fission-function ns
@@ -338,7 +459,7 @@ func (deploy *NewDeploy) fnCreate(fn *fv1.Function, firstcreate bool) (*fscache.
 	// Since newdeploy waits for pods of deployment to be ready,
 	// change the order of kubeObject creation (create service first,
 	// then deployment) to take advantage of waiting time.
-	svc, err := deploy.createOrGetSvc(deployLabels, objName, ns)
+	svc, err := deploy.createOrGetSvc(deployLabels, deployAnnotations, objName, ns)
 	if err != nil {
 		deploy.logger.Error("error creating service", zap.Error(err), zap.String("service", objName))
 		go deploy.cleanupNewdeploy(ns, objName)
@@ -346,14 +467,14 @@ func (deploy *NewDeploy) fnCreate(fn *fv1.Function, firstcreate bool) (*fscache.
 	}
 	svcAddress := fmt.Sprintf("%v.%v", svc.Name, svc.Namespace)
 
-	depl, err := deploy.createOrGetDeployment(fn, env, objName, deployLabels, ns, firstcreate)
+	depl, err := deploy.createOrGetDeployment(fn, env, objName, deployLabels, deployAnnotations, ns)
 	if err != nil {
 		deploy.logger.Error("error creating deployment", zap.Error(err), zap.String("deployment", objName))
 		go deploy.cleanupNewdeploy(ns, objName)
 		return nil, errors.Wrapf(err, "error creating deployment %v", objName)
 	}
 
-	hpa, err := deploy.createOrGetHpa(objName, &fn.Spec.InvokeStrategy.ExecutionStrategy, depl)
+	hpa, err := deploy.createOrGetHpa(objName, &fn.Spec.InvokeStrategy.ExecutionStrategy, depl, deployLabels, deployAnnotations)
 	if err != nil {
 		deploy.logger.Error("error creating HPA", zap.Error(err), zap.String("hpa", objName))
 		go deploy.cleanupNewdeploy(ns, objName)
@@ -394,7 +515,7 @@ func (deploy *NewDeploy) fnCreate(fn *fv1.Function, firstcreate bool) (*fscache.
 		Environment:       env,
 		Address:           svcAddress,
 		KubernetesObjects: kubeObjRefs,
-		Executor:          fscache.NEWDEPLOY,
+		Executor:          fv1.ExecutorTypeNewdeploy,
 	}
 
 	_, err = deploy.fsCache.Add(*fsvc)
@@ -402,6 +523,9 @@ func (deploy *NewDeploy) fnCreate(fn *fv1.Function, firstcreate bool) (*fscache.
 		deploy.logger.Error("error adding function to cache", zap.Error(err), zap.Any("function", fsvc.Function))
 		return fsvc, err
 	}
+
+	deploy.fsCache.IncreaseColdStarts(fn.Metadata.Name, string(fn.Metadata.UID))
+
 	return fsvc, nil
 }
 
@@ -432,7 +556,7 @@ func (deploy *NewDeploy) updateFunction(oldFn *fv1.Function, newFn *fv1.Function
 		deploy.logger.Info("function type changed to new deployment, creating resources",
 			zap.Any("old_function", oldFn.Metadata),
 			zap.Any("new_function", newFn.Metadata))
-		_, err := deploy.createFunction(newFn, true)
+		_, err := deploy.createFunction(newFn)
 		if err != nil {
 			deploy.updateStatus(oldFn, err, "error changing the function's type to newdeploy")
 		}
@@ -544,12 +668,6 @@ func (deploy *NewDeploy) updateFuncDeployment(fn *fv1.Function, env *fv1.Environ
 	deploy.logger.Info("updating deployment due to function/environment update",
 		zap.String("deployment", fnObjName), zap.Any("function", fn.Metadata.Name))
 
-	newDeployment, err := deploy.getDeploymentSpec(fn, env, fnObjName, deployLabels)
-	if err != nil {
-		deploy.updateStatus(fn, err, "failed to get new deployment spec while updating function")
-		return err
-	}
-
 	// to support backward compatibility, if the function was created in default ns, we fall back to creating the
 	// deployment of the function in fission-function ns
 	ns := deploy.namespace
@@ -557,6 +675,22 @@ func (deploy *NewDeploy) updateFuncDeployment(fn *fv1.Function, env *fv1.Environ
 		ns = fn.Metadata.Namespace
 	}
 
+	existingDepl, err := deploy.kubernetesClient.AppsV1().Deployments(ns).Get(fnObjName, metav1.GetOptions{})
+	if err != nil {
+		return err
+	}
+
+	// the resource version inside function packageRef is changed,
+	// so the content of fetchRequest in deployment cmd is different.
+	// Therefore, the deployment update will trigger a rolling update.
+	newDeployment, err := deploy.getDeploymentSpec(fn, env,
+		existingDepl.Spec.Replicas, // use current replicas instead of minscale in the ExecutionStrategy.
+		fnObjName, ns, deployLabels, deploy.getDeployAnnotations(fn.Metadata))
+	if err != nil {
+		deploy.updateStatus(fn, err, "failed to get new deployment spec while updating function")
+		return err
+	}
+
 	err = deploy.updateDeployment(newDeployment, ns)
 	if err != nil {
 		deploy.updateStatus(fn, err, "failed to update deployment while updating function")
@@ -603,19 +737,27 @@ func (deploy *NewDeploy) fnDelete(fn *fv1.Function) error {
 
 // getObjName returns a unique name for kubernetes objects of function
 func (deploy *NewDeploy) getObjName(fn *fv1.Function) string {
-	return strings.ToLower(fmt.Sprintf("newdeploy-%v-%v-%v", fn.Metadata.Name, fn.Metadata.Namespace, uniuri.NewLen(8)))
+	// use meta uuid of function, this ensure we always get the same name for the same function.
+	uid := fn.Metadata.UID[len(fn.Metadata.UID)-17:]
+	return strings.ToLower(fmt.Sprintf("newdeploy-%v-%v-%v", fn.Metadata.Name, fn.Metadata.Namespace, uid))
 }
 
 func (deploy *NewDeploy) getDeployLabels(fnMeta metav1.ObjectMeta, envMeta metav1.ObjectMeta) map[string]string {
+	return map[string]string{
+		types.EXECUTOR_TYPE:         string(fv1.ExecutorTypeNewdeploy),
+		types.ENVIRONMENT_NAME:      envMeta.Name,
+		types.ENVIRONMENT_NAMESPACE: envMeta.Namespace,
+		types.ENVIRONMENT_UID:       string(envMeta.UID),
+		types.FUNCTION_NAME:         fnMeta.Name,
+		types.FUNCTION_NAMESPACE:    fnMeta.Namespace,
+		types.FUNCTION_UID:          string(fnMeta.UID),
+	}
+}
+
+func (deploy *NewDeploy) getDeployAnnotations(fnMeta metav1.ObjectMeta) map[string]string {
 	return map[string]string{
 		types.EXECUTOR_INSTANCEID_LABEL: deploy.instanceID,
-		types.EXECUTOR_TYPE:             fv1.ExecutorTypeNewdeploy,
+		types.FUNCTION_RESOURCE_VERSION: fnMeta.ResourceVersion,
-		types.ENVIRONMENT_NAME:          envMeta.Name,
-		types.ENVIRONMENT_NAMESPACE:     envMeta.Namespace,
-		types.ENVIRONMENT_UID:           string(envMeta.UID),
-		types.FUNCTION_NAME:             fnMeta.Name,
-		types.FUNCTION_NAMESPACE:        fnMeta.Namespace,
-		types.FUNCTION_UID:              string(fnMeta.UID),
 	}
 }
 
@@ -625,42 +767,6 @@ func (deploy *NewDeploy) updateStatus(fn *fv1.Function, err error, message strin
 	deploy.logger.Error("function status update", zap.Error(err), zap.Any("function", fn), zap.String("message", message))
 }
 
-// IsValid does a get on the service address to ensure it's a valid service, then
-// scale deployment to 1 replica if there are no available replicas for function.
-// Return true if no error occurs, return false otherwise.
-func (deploy *NewDeploy) IsValid(fsvc *fscache.FuncSvc) bool {
-	service := strings.Split(fsvc.Address, ".")
-	if len(service) == 0 {
-		return false
-	}
-
-	_, err := deploy.kubernetesClient.CoreV1().Services(service[1]).Get(service[0], metav1.GetOptions{})
-	if err != nil {
-		deploy.logger.Error("error validating function service address", zap.String("function", fsvc.Function.Name), zap.Error(err))
-		return false
-	}
-
-	deployObj := getDeploymentObj(fsvc.KubernetesObjects)
-	if deployObj == nil {
-		deploy.logger.Error("deployment obj for function does not exist", zap.String("function", fsvc.Function.Name))
-		return false
-	}
-
-	currentDeploy, err := deploy.kubernetesClient.AppsV1().
-		Deployments(deployObj.Namespace).Get(deployObj.Name, metav1.GetOptions{})
-	if err != nil {
-		deploy.logger.Error("error validating function deployment", zap.Error(err), zap.String("function", fsvc.Function.Name))
-		return false
-	}
-
-	// return directly when available replicas > 0
-	if currentDeploy.Status.AvailableReplicas > 0 {
-		return true
-	}
-
-	return false
-}
-
 // idleObjectReaper reaps objects after certain idle time
 func (deploy *NewDeploy) idleObjectReaper() {
 
@@ -685,7 +791,7 @@ func (deploy *NewDeploy) idleObjectReaper() {
 		}
 
 		for _, fsvc := range funcSvcs {
-			if fsvc.Executor != fscache.NEWDEPLOY {
+			if fsvc.Executor != fv1.ExecutorTypeNewdeploy {
 				continue
 			}
 
@@ -701,7 +807,7 @@ func (deploy *NewDeploy) idleObjectReaper() {
 			if err != nil {
 				// Newdeploy manager handles the function delete event and clean cache/kubeobjs itself,
 				// so we ignore the not found error for functions with newdeploy executor type here.
-				if k8sErrs.IsNotFound(err) && fsvc.Executor == fscache.NEWDEPLOY {
+				if k8sErrs.IsNotFound(err) && fsvc.Executor == fv1.ExecutorTypeNewdeploy {
 					continue
 				}
 				deploy.logger.Error("error getting function", zap.Error(err), zap.String("function", fsvc.Function.Name))
@@ -717,7 +823,7 @@ func (deploy *NewDeploy) idleObjectReaper() {
 			currentDeploy, err := deploy.kubernetesClient.AppsV1().
 				Deployments(deployObj.Namespace).Get(deployObj.Name, metav1.GetOptions{})
 			if err != nil {
-				deploy.logger.Error("error validating function deployment", zap.Error(err), zap.String("function", fsvc.Function.Name))
+				deploy.logger.Error("error getting function deployment", zap.Error(err), zap.String("function", fsvc.Function.Name))
 				continue
 			}
 

pkg/executor/executortype/poolmgr/gp.go

@@ -33,8 +33,10 @@ import (
 	"go.uber.org/zap"
 	appsv1 "k8s.io/api/apps/v1"
 	apiv1 "k8s.io/api/core/v1"
+	k8sErrs "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/labels"
+	k8sTypes "k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/intstr"
 	"k8s.io/client-go/kubernetes"
 
@@ -64,9 +66,9 @@ type (
 		kubernetesClient       *kubernetes.Clientset
 		fissionClient          *crd.FissionClient
 		instanceId             string // poolmgr instance id
-		labelsForPool          map[string]string
 		requestChannel         chan *choosePodRequest
 		fetcherConfig          *fetcherConfig.Config
+		stopCh                 context.CancelFunc
 	}
 
 	// serialize the choosing of pods so that choices don't conflict
@@ -97,6 +99,8 @@ func MakeGenericPool(
 
 	gpLogger.Info("creating pool", zap.Any("environment", env.Metadata))
 
+	ctx, stopCh := context.WithCancel(context.Background())
+
 	// TODO: in general we need to provide the user a way to configure pools.  Initial
 	// replicas, autoscaling params, various timeouts, etc.
 	gp := &GenericPool{
@@ -116,6 +120,7 @@ func MakeGenericPool(
 		instanceId:        instanceId,
 		useSvc:            false,       // defaults off -- svc takes a second or more to become routable, slowing cold start
 		useIstio:          enableIstio, // defaults off -- istio integration requires pod relabeling and it takes a second or more to become routable, slowing cold start
+		stopCh:            stopCh,
 	}
 
 	gp.runtimeImagePullPolicy = utils.GetImagePullPolicy(os.Getenv("RUNTIME_IMAGE_PULL_POLICY"))
@@ -127,7 +132,7 @@ func MakeGenericPool(
 	}
 
 	// Labels for generic deployment/RS/pods.
-	gp.labelsForPool = gp.getDeployLabels()
+	//gp.labelsForPool = gp.getDeployLabels()
 
 	// create the pool
 	err = gp.createPool()
@@ -136,31 +141,41 @@ func MakeGenericPool(
 	}
 	gpLogger.Info("deployment created", zap.Any("environment", env.Metadata))
 
-	go gp.choosePodService()
+	go gp.choosePodService(ctx)
 
 	return gp, nil
 }
 
-func (gp *GenericPool) getDeployLabels() map[string]string {
+func (gp *GenericPool) getEnvironmentPoolLabels() map[string]string {
+	return map[string]string{
+		types.EXECUTOR_TYPE:         string(fv1.ExecutorTypePoolmgr),
+		types.ENVIRONMENT_NAME:      gp.env.Metadata.Name,
+		types.ENVIRONMENT_NAMESPACE: gp.env.Metadata.Namespace,
+		types.ENVIRONMENT_UID:       string(gp.env.Metadata.UID),
+		"managed":                   "true", // this allows us to easily find pods managed by the deployment
+	}
+}
+
+func (gp *GenericPool) getDeployAnnotations() map[string]string {
 	return map[string]string{
 		fv1.EXECUTOR_INSTANCEID_LABEL: gp.instanceId,
-		types.EXECUTOR_TYPE:           fv1.ExecutorTypePoolmgr,
-		types.ENVIRONMENT_NAME:        gp.env.Metadata.Name,
-		types.ENVIRONMENT_NAMESPACE:   gp.env.Metadata.Namespace,
-		types.ENVIRONMENT_UID:         string(gp.env.Metadata.UID),
-		"managed":                     "true", // this allows us to easily find pods managed by the deployment
 	}
 }
 
 // choosePodService serializes the choosing of pods
-func (gp *GenericPool) choosePodService() {
+func (gp *GenericPool) choosePodService(ctx context.Context) {
-	for req := range gp.requestChannel {
+	for {
-		pod, err := gp._choosePod(req.newLabels)
+		select {
-		if err != nil {
+		case req := <-gp.requestChannel:
-			req.responseChannel <- &choosePodResponse{error: err}
+			pod, err := gp._choosePod(req.newLabels)
-			continue
+			if err != nil {
+				req.responseChannel <- &choosePodResponse{error: err}
+				continue
+			}
+			req.responseChannel <- &choosePodResponse{pod: pod}
+		case <-ctx.Done():
+			return
 		}
-		req.responseChannel <- &choosePodResponse{pod: pod}
 	}
 }
 
@@ -231,6 +246,11 @@ func (gp *GenericPool) _choosePod(newLabels map[string]string) (*apiv1.Pod, erro
 			// modified, this should fail; in that case just
 			// retry.
 			chosenPod.ObjectMeta.Labels = newLabels
+
+			// Append executor instance id to pod annotations to
+			// indicate this pod is managed by this executor.
+			chosenPod.ObjectMeta.Annotations = gp.getDeployAnnotations()
+
 			_, err = gp.kubernetesClient.CoreV1().Pods(gp.namespace).Update(chosenPod)
 			if err != nil {
 				gp.logger.Error("failed to relabel pod", zap.Error(err), zap.String("pod", chosenPod.ObjectMeta.Name))
@@ -244,7 +264,7 @@ func (gp *GenericPool) _choosePod(newLabels map[string]string) (*apiv1.Pod, erro
 }
 
 func (gp *GenericPool) labelsForFunction(metadata *metav1.ObjectMeta) map[string]string {
-	label := gp.getDeployLabels()
+	label := gp.getEnvironmentPoolLabels()
 	label[types.FUNCTION_NAME] = metadata.Name
 	label[types.FUNCTION_UID] = string(metadata.UID)
 	label[types.FUNCTION_NAMESPACE] = metadata.Namespace // function CRD must stay within same namespace of environment CRD
@@ -259,11 +279,6 @@ func (gp *GenericPool) scheduleDeletePod(name string) {
 		// cleaned up.  (We need a better solutions for both those things; log
 		// aggregation and storage will help.)
 		gp.logger.Error("error in pod - scheduling cleanup", zap.String("pod", name))
-		// Ignore sleep here if istio feature is enabled, function pod
-		// will be deleted after 6 mins (terminationGracePeriodSeconds).
-		if !gp.useIstio {
-			time.Sleep(5 * time.Minute)
-		}
 		gp.kubernetesClient.CoreV1().Pods(gp.namespace).Delete(name, nil)
 	}()
 }
@@ -330,12 +345,15 @@ func (gp *GenericPool) specializePod(ctx context.Context, pod *apiv1.Pod, fn *fv
 
 // getPoolName returns a unique name of an environment
 func (gp *GenericPool) getPoolName() string {
-	return strings.ToLower(fmt.Sprintf("poolmgr-%v-%v-%v", gp.env.Metadata.Name, gp.env.Metadata.Namespace, uniuri.NewLen(8)))
+	return strings.ToLower(fmt.Sprintf("poolmgr-%v-%v-%v", gp.env.Metadata.Name, gp.env.Metadata.Namespace, gp.env.Metadata.ResourceVersion))
 }
 
 // A pool is a deployment of generic containers for an env.  This
 // creates the pool but doesn't wait for any pods to be ready.
 func (gp *GenericPool) createPool() error {
+	deployLabels := gp.getEnvironmentPoolLabels()
+	deployAnnotations := gp.getDeployAnnotations()
+
 	// Use long terminationGracePeriodSeconds for connection draining in case that
 	// pod still runs user functions.
 	gracePeriodSeconds := int64(6 * 60)
@@ -347,6 +365,12 @@ func (gp *GenericPool) createPool() error {
 	if podAnnotations == nil {
 		podAnnotations = make(map[string]string)
 	}
+
+	// Here, we don't append executor instance-id to pod annotations
+	// to prevent unwanted rolling updates occur. Pool manager will
+	// append executor instance-id to pod annotations when a pod is chosen
+	// for function specialization.
+
 	if gp.useIstio && gp.env.Spec.AllowAccessToExternalNetwork {
 		podAnnotations["sidecar.istio.io/inject"] = "false"
 	}
@@ -389,30 +413,35 @@ func (gp *GenericPool) createPool() error {
 		return err
 	}
 
+	pod := apiv1.PodTemplateSpec{
+		ObjectMeta: metav1.ObjectMeta{
+			Labels:      deployLabels,
+			Annotations: podAnnotations,
+		},
+		Spec: apiv1.PodSpec{
+			Containers:         []apiv1.Container{*container},
+			ServiceAccountName: "fission-fetcher",
+			// TerminationGracePeriodSeconds should be equal to the
+			// sleep time of preStop to make sure that SIGTERM is sent
+			// to pod after 6 mins.
+			TerminationGracePeriodSeconds: &gracePeriodSeconds,
+		},
+	}
+
+	pod.Spec = *(util.ApplyImagePullSecret(gp.env.Spec.ImagePullSecret, pod.Spec))
+
 	deployment := &appsv1.Deployment{
 		ObjectMeta: metav1.ObjectMeta{
-			Name:   gp.getPoolName(),
+			Name:        gp.getPoolName(),
-			Labels: gp.labelsForPool,
+			Labels:      deployLabels,
+			Annotations: deployAnnotations,
 		},
 		Spec: appsv1.DeploymentSpec{
 			Replicas: &gp.replicas,
 			Selector: &metav1.LabelSelector{
-				MatchLabels: gp.labelsForPool,
+				MatchLabels: deployLabels,
-			},
-			Template: apiv1.PodTemplateSpec{
-				ObjectMeta: metav1.ObjectMeta{
-					Labels:      gp.labelsForPool,
-					Annotations: podAnnotations,
-				},
-				Spec: apiv1.PodSpec{
-					Containers:         []apiv1.Container{*container},
-					ServiceAccountName: "fission-fetcher",
-					// TerminationGracePeriodSeconds should be equal to the
-					// sleep time of preStop to make sure that SIGTERM is sent
-					// to pod after 6 mins.
-					TerminationGracePeriodSeconds: &gracePeriodSeconds,
-				},
 			},
+			Template: pod,
 		},
 	}
 
@@ -430,11 +459,25 @@ func (gp *GenericPool) createPool() error {
 		deployment.Spec.Template.Spec = *newPodSpec
 	}
 
-	depl, err := gp.kubernetesClient.AppsV1().Deployments(gp.namespace).Create(deployment)
+	depl, err := gp.kubernetesClient.AppsV1().Deployments(gp.namespace).Get(deployment.Name, metav1.GetOptions{})
+	if err == nil {
+		if depl.Annotations[fv1.EXECUTOR_INSTANCEID_LABEL] != gp.instanceId {
+			deployment.Annotations[fv1.EXECUTOR_INSTANCEID_LABEL] = gp.instanceId
+			depl, err = gp.kubernetesClient.AppsV1().Deployments(gp.namespace).Update(deployment)
+		}
+		gp.deployment = depl
+		return err
+	} else if !k8sErrs.IsNotFound(err) {
+		gp.logger.Error("error getting deployment in kubernetes", zap.Error(err), zap.String("deployment", deployment.Name))
+		return err
+	}
+
+	depl, err = gp.kubernetesClient.AppsV1().Deployments(gp.namespace).Create(deployment)
 	if err != nil {
 		gp.logger.Error("error creating deployment in kubernetes", zap.Error(err), zap.String("deployment", deployment.Name))
 		return err
 	}
+
 	gp.deployment = depl
 	return nil
 }
@@ -467,14 +510,17 @@ func (gp *GenericPool) waitForReadyPod() error {
 
 			// Since even single pod is not ready, choosing the first pod to inspect is a good approximation. In future this can be done better
 			pod := podList.Items[0]
-			multierr := &multierror.Error{}
+			errs := &multierror.Error{}
 			for _, cStatus := range pod.Status.ContainerStatuses {
 				if !cStatus.Ready {
-					multierr = multierror.Append(multierr, errors.New(fmt.Sprintf("%v: %v", cStatus.State.Waiting.Reason, cStatus.State.Waiting.Message)))
+					errs = multierror.Append(errs, errors.New(fmt.Sprintf("%v: %v", cStatus.State.Waiting.Reason, cStatus.State.Waiting.Message)))
 				}
 			}
-			return errors.Wrapf(multierr, "Timeout: waited too long for pod of deployment %v in namespace %v to be ready",
+			if errs.ErrorOrNil() != nil {
-				gp.deployment.ObjectMeta.Name, gp.namespace)
+				return errors.Wrapf(errs, "Timeout: waited too long for pod of deployment %v in namespace %v to be ready",
+					gp.deployment.ObjectMeta.Name, gp.namespace)
+			}
+			return nil
 		}
 		time.Sleep(1000 * time.Millisecond)
 	}
@@ -483,14 +529,15 @@ func (gp *GenericPool) waitForReadyPod() error {
 func (gp *GenericPool) createSvc(name string, labels map[string]string) (*apiv1.Service, error) {
 	service := apiv1.Service{
 		ObjectMeta: metav1.ObjectMeta{
-			Name: name,
+			Name:   name,
+			Labels: labels,
 		},
 		Spec: apiv1.ServiceSpec{
 			Type: apiv1.ServiceTypeClusterIP,
 			Ports: []apiv1.ServicePort{
 				{
 					Protocol:   apiv1.ProtocolTCP,
-					Port:       80,
+					Port:       8888,
 					TargetPort: intstr.FromInt(8888),
 				},
 			},
@@ -503,7 +550,7 @@ func (gp *GenericPool) createSvc(name string, labels map[string]string) (*apiv1.
 
 func (gp *GenericPool) getFuncSvc(ctx context.Context, fn *fv1.Function) (*fscache.FuncSvc, error) {
 	gp.logger.Info("choosing pod from pool", zap.Any("function", fn.Metadata))
-	newLabels := gp.labelsForFunction(&fn.Metadata)
+	funcLabels := gp.labelsForFunction(&fn.Metadata)
 
 	if gp.useIstio {
 		// Istio only allows accessing pod through k8s service, and requests come to
@@ -546,7 +593,7 @@ func (gp *GenericPool) getFuncSvc(ctx context.Context, fn *fv1.Function) (*fscac
 		}
 	}
 
-	pod, err := gp.choosePod(newLabels)
+	pod, err := gp.choosePod(funcLabels)
 	if err != nil {
 		return nil, err
 	}
@@ -565,7 +612,7 @@ func (gp *GenericPool) getFuncSvc(ctx context.Context, fn *fv1.Function) (*fscac
 			svcName = fmt.Sprintf("%s-%v", svcName, fn.Metadata.UID)
 		}
 
-		svc, err := gp.createSvc(svcName, newLabels)
+		svc, err := gp.createSvc(svcName, funcLabels)
 		if err != nil {
 			gp.scheduleDeletePod(pod.ObjectMeta.Name)
 			return nil, err
@@ -577,7 +624,7 @@ func (gp *GenericPool) getFuncSvc(ctx context.Context, fn *fv1.Function) (*fscac
 
 		// the fission router isn't in the same namespace, so return a
 		// namespace-qualified hostname
-		svcHost = fmt.Sprintf("%v.%v", svcName, gp.namespace)
+		svcHost = fmt.Sprintf("%v.%v:8888", svcName, gp.namespace)
 	} else if gp.useIstio {
 		svc := utils.GetFunctionIstioServiceName(fn.Metadata.Name, fn.Metadata.Namespace)
 		svcHost = fmt.Sprintf("%v.%v:8888", svc, gp.namespace)
@@ -585,6 +632,18 @@ func (gp *GenericPool) getFuncSvc(ctx context.Context, fn *fv1.Function) (*fscac
 		svcHost = fmt.Sprintf("%v:8888", pod.Status.PodIP)
 	}
 
+	// patch svc-host and resource version to the pod annotations for new executor to adopt the pod
+	patch := fmt.Sprintf(`{"metadata":{"annotations":{"%v":"%v","%v":"%v"}}}`,
+		types.ANNOTATION_SVC_HOST, svcHost, types.FUNCTION_RESOURCE_VERSION, fn.Metadata.ResourceVersion)
+	p, err := gp.kubernetesClient.CoreV1().Pods(pod.Namespace).Patch(pod.Name, k8sTypes.StrategicMergePatchType, []byte(patch))
+	if err != nil {
+		// just log the error since it won't affect the function serving
+		gp.logger.Warn("error patching svc-host to pod", zap.Error(err),
+			zap.String("pod", pod.Name), zap.String("ns", pod.Namespace))
+	} else {
+		pod = p
+	}
+
 	gp.logger.Info("specialized pod",
 		zap.String("pod", pod.ObjectMeta.Name),
 		zap.String("podNamespace", pod.ObjectMeta.Namespace),
@@ -610,7 +669,7 @@ func (gp *GenericPool) getFuncSvc(ctx context.Context, fn *fv1.Function) (*fscac
 		Environment:       gp.env,
 		Address:           svcHost,
 		KubernetesObjects: kubeObjRefs,
-		Executor:          fscache.POOLMGR,
+		Executor:          fv1.ExecutorTypePoolmgr,
 		Ctime:             time.Now(),
 		Atime:             time.Now(),
 	}
@@ -619,15 +678,21 @@ func (gp *GenericPool) getFuncSvc(ctx context.Context, fn *fv1.Function) (*fscac
 	if err != nil {
 		return nil, err
 	}
+
+	gp.fsCache.IncreaseColdStarts(fn.Metadata.Name, string(fn.Metadata.UID))
+
 	return fsvc, nil
 }
 
 // destroys the pool -- the deployment, replicaset and pods
 func (gp *GenericPool) destroy() error {
+	gp.stopCh()
+
 	deletePropagation := metav1.DeletePropagationBackground
 	delOpt := metav1.DeleteOptions{
 		PropagationPolicy: &deletePropagation,
 	}
+
 	err := gp.kubernetesClient.AppsV1().
 		Deployments(gp.namespace).Delete(gp.deployment.ObjectMeta.Name, &delOpt)
 	if err != nil {

pkg/executor/executortype/poolmgr/gpm.go

@@ -18,13 +18,17 @@ package poolmgr
 
 import (
 	"context"
+	"fmt"
+	"math/rand"
 	"os"
 	"strconv"
 	"strings"
+	"sync"
 	"time"
 
-	"github.com/fission/fission/pkg/utils"
+	"github.com/hashicorp/go-multierror"
 	"go.uber.org/zap"
+	apiv1 "k8s.io/api/core/v1"
 	k8serrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/labels"
@@ -35,12 +39,16 @@ import (
 	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
 	"github.com/fission/fission/pkg/cache"
 	"github.com/fission/fission/pkg/crd"
+	"github.com/fission/fission/pkg/executor/executortype"
 	"github.com/fission/fission/pkg/executor/fscache"
 	"github.com/fission/fission/pkg/executor/reaper"
 	fetcherConfig "github.com/fission/fission/pkg/fetcher/config"
 	"github.com/fission/fission/pkg/types"
+	"github.com/fission/fission/pkg/utils"
 )
 
+var _ executortype.ExecutorType = &GenericPoolManager{}
+
 type requestType int
 
 const (
@@ -90,7 +98,7 @@ func MakeGenericPoolManager(
 	kubernetesClient *kubernetes.Clientset,
 	functionNamespace string,
 	fetcherConfig *fetcherConfig.Config,
-	instanceId string) *GenericPoolManager {
+	instanceId string) executortype.ExecutorType {
 
 	gpmLogger := logger.Named("generic_pool_manager")
 
@@ -107,6 +115,7 @@ func MakeGenericPoolManager(
 		idlePodReapTime:  2 * time.Minute,
 		fetcherConfig:    fetcherConfig,
 	}
+
 	go gpm.service()
 	go gpm.eagerPoolCreator()
 
@@ -132,6 +141,71 @@ func (gpm *GenericPoolManager) Run(ctx context.Context) {
 	go gpm.idleObjectReaper()
 }
 
+func (gpm *GenericPoolManager) GetTypeName() fv1.ExecutorType {
+	return fv1.ExecutorTypePoolmgr
+}
+
+func (gpm *GenericPoolManager) GetFuncSvc(ctx context.Context, fn *fv1.Function) (*fscache.FuncSvc, error) {
+	// from Func -> get Env
+	gpm.logger.Debug("getting environment for function", zap.String("function", fn.Metadata.Name))
+	env, err := gpm.getFunctionEnv(fn)
+	if err != nil {
+		return nil, err
+	}
+
+	pool, err := gpm.getPool(env)
+	if err != nil {
+		return nil, err
+	}
+
+	// from GenericPool -> get one function container
+	// (this also adds to the cache)
+	gpm.logger.Debug("getting function service from pool", zap.String("function", fn.Metadata.Name))
+	return pool.getFuncSvc(ctx, fn)
+}
+
+func (gpm *GenericPoolManager) GetFuncSvcFromCache(fn *fv1.Function) (*fscache.FuncSvc, error) {
+	return gpm.fsCache.GetByFunction(&fn.Metadata)
+}
+
+func (gpm *GenericPoolManager) DeleteFuncSvcFromCache(fsvc *fscache.FuncSvc) {
+	gpm.fsCache.DeleteEntry(fsvc)
+}
+
+func (gpm *GenericPoolManager) TapService(svcHost string) error {
+	err := gpm.fsCache.TouchByAddress(svcHost)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+// IsValid checks if pod is not deleted and that it has the address passed as the argument. Also checks that all the
+// containers in it are reporting a ready status for the healthCheck.
+func (gpm *GenericPoolManager) IsValid(fsvc *fscache.FuncSvc) bool {
+	for _, obj := range fsvc.KubernetesObjects {
+		if strings.ToLower(obj.Kind) == "pod" {
+			pod, err := gpm.kubernetesClient.CoreV1().Pods(obj.Namespace).Get(obj.Name, metav1.GetOptions{})
+			if err == nil && utils.IsReadyPod(pod) {
+				// Normally, the address format is http://[pod-ip]:[port], however, if the
+				// Istio is enabled the address format changes to http://[svc-name]:[port].
+				// So if the Istio is enabled and pod is in ready state, we return true directly;
+				// Otherwise, we need to ensure that the address contains pod ip.
+				if gpm.enableIstio ||
+					(!gpm.enableIstio && strings.Contains(fsvc.Address, pod.Status.PodIP)) {
+					gpm.logger.Debug("valid address",
+						zap.String("address", fsvc.Address),
+						zap.Any("function", fsvc.Function),
+						zap.String("executor", string(fsvc.Executor)),
+					)
+					return true
+				}
+			}
+		}
+	}
+	return false
+}
+
 func (gpm *GenericPoolManager) RefreshFuncPods(logger *zap.Logger, f fv1.Function) error {
 
 	env, err := gpm.fissionClient.Environments(f.Spec.Environment.Namespace).Get(f.Spec.Environment.Name)
@@ -174,6 +248,159 @@ func (gpm *GenericPoolManager) RefreshFuncPods(logger *zap.Logger, f fv1.Functio
 	return nil
 }
 
+func (gpm *GenericPoolManager) AdoptExistingResources() {
+	envs, err := gpm.fissionClient.Environments(metav1.NamespaceAll).List(metav1.ListOptions{})
+	if err != nil {
+		gpm.logger.Error("error getting environment list", zap.Error(err))
+		return
+	}
+
+	envMap := make(map[string]fv1.Environment, len(envs.Items))
+	wg := &sync.WaitGroup{}
+
+	for i := range envs.Items {
+		env := envs.Items[i]
+
+		if gpm.getEnvPoolsize(&env) > 0 {
+			wg.Add(1)
+			go func() {
+				defer wg.Done()
+				_, err := gpm.getPool(&env)
+				if err != nil {
+					gpm.logger.Error("adopt pool failed", zap.Error(err))
+				}
+			}()
+		}
+
+		// create environment map for later use
+		key := fmt.Sprintf("%v/%v", env.Metadata.Namespace, env.Metadata.Name)
+		envMap[key] = env
+	}
+
+	l := map[string]string{
+		types.EXECUTOR_TYPE: string(fv1.ExecutorTypePoolmgr),
+	}
+
+	podList, err := gpm.kubernetesClient.CoreV1().Pods(metav1.NamespaceAll).List(metav1.ListOptions{
+		LabelSelector: labels.Set(l).AsSelector().String(),
+	})
+
+	if err != nil {
+		gpm.logger.Error("error getting pod list", zap.Error(err))
+		return
+	}
+
+	for i := range podList.Items {
+		pod := &podList.Items[i]
+		if !utils.IsReadyPod(pod) {
+			continue
+		}
+
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+
+			// avoid too many requests arrive Kubernetes API server at the same time.
+			time.Sleep(time.Duration(rand.Intn(30)) * time.Millisecond)
+
+			patch := fmt.Sprintf(`{"metadata":{"annotations":{"%v":"%v"}}}`, types.EXECUTOR_INSTANCEID_LABEL, gpm.instanceId)
+			pod, err = gpm.kubernetesClient.CoreV1().Pods(pod.Namespace).Patch(pod.Name, k8sTypes.StrategicMergePatchType, []byte(patch))
+			if err != nil {
+				// just log the error since it won't affect the function serving
+				gpm.logger.Warn("error patching executor instance ID of pod", zap.Error(err),
+					zap.String("pod", pod.Name), zap.String("ns", pod.Namespace))
+				return
+			}
+
+			// for unspecialized pod, we only update its annotations
+			if pod.Labels["managed"] == "true" {
+				return
+			}
+
+			fnName, ok1 := pod.Labels[types.FUNCTION_NAME]
+			fnNS, ok2 := pod.Labels[types.FUNCTION_NAMESPACE]
+			fnUID, ok3 := pod.Labels[types.FUNCTION_UID]
+			fnRV, ok4 := pod.Annotations[types.FUNCTION_RESOURCE_VERSION]
+			envName, ok5 := pod.Labels[types.ENVIRONMENT_NAME]
+			envNS, ok6 := pod.Labels[types.ENVIRONMENT_NAMESPACE]
+			svcHost, ok7 := pod.Annotations[types.ANNOTATION_SVC_HOST]
+			env, ok8 := envMap[fmt.Sprintf("%v/%v", envNS, envName)]
+
+			if !(ok1 && ok2 && ok3 && ok4 && ok5 && ok6 && ok7 && ok8) {
+				gpm.logger.Warn("failed to adopt pod for function due to lack of necessary information",
+					zap.String("pod", pod.Name), zap.Any("labels", pod.Labels), zap.Any("annotations", pod.Annotations),
+					zap.String("env", env.Metadata.Name))
+				return
+			}
+
+			fsvc := fscache.FuncSvc{
+				Name: pod.Name,
+				Function: &metav1.ObjectMeta{
+					Name:            fnName,
+					Namespace:       fnNS,
+					UID:             k8sTypes.UID(fnUID),
+					ResourceVersion: fnRV,
+				},
+				Environment: &env,
+				Address:     svcHost,
+				KubernetesObjects: []apiv1.ObjectReference{
+					{
+						Kind:            "pod",
+						Name:            pod.Name,
+						APIVersion:      pod.TypeMeta.APIVersion,
+						Namespace:       pod.ObjectMeta.Namespace,
+						ResourceVersion: pod.ObjectMeta.ResourceVersion,
+						UID:             pod.ObjectMeta.UID,
+					},
+				},
+				Executor: fv1.ExecutorTypePoolmgr,
+				Ctime:    time.Now(),
+				Atime:    time.Now(),
+			}
+
+			_, err = gpm.fsCache.Add(fsvc)
+			if err != nil {
+				// If fsvc already exists we just skip the duplicate one. And let reaper to recycle the duplicate pods.
+				// This is for the case that there are multiple function pods for the same function due to unknown reason.
+				if !fscache.IsNameExistError(err) {
+					gpm.logger.Warn("failed to adopt pod for function", zap.Error(err), zap.String("pod", pod.Name))
+				}
+
+				return
+			}
+
+			gpm.logger.Info("adopt function pod",
+				zap.String("pod", pod.Name), zap.Any("labels", pod.Labels), zap.Any("annotations", pod.Annotations))
+		}()
+	}
+
+	wg.Wait()
+}
+
+func (gpm *GenericPoolManager) CleanupOldExecutorObjects() {
+	gpm.logger.Info("Poolmanager starts to clean orphaned resources", zap.String("instanceID", gpm.instanceId))
+
+	errs := &multierror.Error{}
+	listOpts := metav1.ListOptions{
+		LabelSelector: labels.Set(map[string]string{types.EXECUTOR_TYPE: string(fv1.ExecutorTypePoolmgr)}).AsSelector().String(),
+	}
+
+	err := reaper.CleanupDeployments(gpm.logger, gpm.kubernetesClient, gpm.instanceId, listOpts)
+	if err != nil {
+		errs = multierror.Append(errs, err)
+	}
+
+	err = reaper.CleanupPods(gpm.logger, gpm.kubernetesClient, gpm.instanceId, listOpts)
+	if err != nil {
+		errs = multierror.Append(errs, err)
+	}
+
+	if errs.ErrorOrNil() != nil {
+		// TODO retry reaper; logged and ignored for now
+		gpm.logger.Error("Failed to cleanup old executor objects", zap.Error(err))
+	}
+}
+
 func (gpm *GenericPoolManager) service() {
 	for {
 		req := <-gpm.requestChannel
@@ -246,25 +473,6 @@ func (gpm *GenericPoolManager) cleanupPools(envs []fv1.Environment) {
 	}
 }
 
-func (gpm *GenericPoolManager) GetFuncSvc(ctx context.Context, fn *fv1.Function) (*fscache.FuncSvc, error) {
-	// from Func -> get Env
-	gpm.logger.Debug("getting environment for function", zap.String("function", fn.Metadata.Name))
-	env, err := gpm.getFunctionEnv(fn)
-	if err != nil {
-		return nil, err
-	}
-
-	pool, err := gpm.getPool(env)
-	if err != nil {
-		return nil, err
-	}
-
-	// from GenericPool -> get one function container
-	// (this also adds to the cache)
-	gpm.logger.Debug("getting function service from pool", zap.String("function", fn.Metadata.Name))
-	return pool.getFuncSvc(ctx, fn)
-}
-
 func (gpm *GenericPoolManager) getFunctionEnv(fn *fv1.Function) (*fv1.Environment, error) {
 	var env *fv1.Environment
 
@@ -297,29 +505,38 @@ func (gpm *GenericPoolManager) eagerPoolCreator() {
 		if err != nil {
 			if utils.IsNetworkError(err) {
 				gpm.logger.Error("encountered network error, retrying", zap.Error(err))
-				time.Sleep(5 * time.Second)
+			} else {
-				continue
+				gpm.logger.Error("failed to get environment list", zap.Error(err))
 			}
-			gpm.logger.Error("failed to get environment list", zap.Error(err))
+			time.Sleep(5 * time.Second)
+			continue
 		}
 
 		// Create pools for all envs.  TODO: we should make this a bit less eager, only
 		// creating pools for envs that are actually used by functions.  Also we might want
 		// to keep these eagerly created pools smaller than the ones created when there are
 		// actual function calls.
+
+		wg := &sync.WaitGroup{}
+
 		for i := range envs.Items {
 			env := envs.Items[i]
 			// Create pool only if poolsize greater than zero
 			if gpm.getEnvPoolsize(&env) > 0 {
-				_, err := gpm.getPool(&envs.Items[i])
+				wg.Add(1)
-				if err != nil {
+				go func() {
-					gpm.logger.Error("eager-create pool failed", zap.Error(err))
+					defer wg.Done()
-				}
+					_, err := gpm.getPool(&env)
+					if err != nil {
+						gpm.logger.Error("eager-create pool failed", zap.Error(err))
+					}
+				}()
 			}
 		}
 
 		// Clean up pools whose env was deleted
 		gpm.cleanupPools(envs.Items)
+		wg.Wait()
 		time.Sleep(pollSleep)
 	}
 }
@@ -334,28 +551,6 @@ func (gpm *GenericPoolManager) getEnvPoolsize(env *fv1.Environment) int32 {
 	return poolsize
 }
 
-// IsValid checks if pod is not deleted and that it has the address passed as the argument. Also checks that all the
-// containers in it are reporting a ready status for the healthCheck.
-func (gpm *GenericPoolManager) IsValid(fsvc *fscache.FuncSvc) bool {
-	for _, obj := range fsvc.KubernetesObjects {
-		if obj.Kind == "pod" {
-			pod, err := gpm.kubernetesClient.CoreV1().Pods(obj.Namespace).Get(obj.Name, metav1.GetOptions{})
-			if err == nil && utils.IsReadyPod(pod) {
-				// Normally, the address format is http://[pod-ip]:[port], however, if the
-				// Istio is enabled the address format changes to http://[svc-name]:[port].
-				// So if the Istio is enabled and pod is in ready state, we return true directly;
-				// Otherwise, we need to ensure that the address contains pod ip.
-				if gpm.enableIstio ||
-					(!gpm.enableIstio && strings.Contains(fsvc.Address, pod.Status.PodIP)) {
-					gpm.logger.Debug("valid address", zap.String("address", fsvc.Address))
-					return true
-				}
-			}
-		}
-	}
-	return false
-}
-
 // idleObjectReaper reaps objects after certain idle time
 func (gpm *GenericPoolManager) idleObjectReaper() {
 
@@ -379,8 +574,10 @@ func (gpm *GenericPoolManager) idleObjectReaper() {
 			continue
 		}
 
-		for _, fsvc := range funcSvcs {
+		for i := range funcSvcs {
-			if fsvc.Executor != fscache.POOLMGR {
+			fsvc := funcSvcs[i]
+
+			if fsvc.Executor != fv1.ExecutorTypePoolmgr {
 				continue
 			}
 
@@ -396,20 +593,26 @@ func (gpm *GenericPoolManager) idleObjectReaper() {
 				continue
 			}
 
-			deleted, err := gpm.fsCache.DeleteOld(fsvc, gpm.idlePodReapTime)
+			go func() {
-			if err != nil {
+				deleted, err := gpm.fsCache.DeleteOld(fsvc, gpm.idlePodReapTime)
-				gpm.logger.Error("error deleting Kubernetes objects for function service",
+				if err != nil {
-					zap.Error(err),
+					gpm.logger.Error("error deleting Kubernetes objects for function service",
-					zap.Any("service", fsvc))
+						zap.Error(err),
-			}
+						zap.Any("service", fsvc))
-
+				}
-			if !deleted {
+				if deleted {
-				continue
+					for i := range fsvc.KubernetesObjects {
-			}
+						gpm.logger.Info("release idle function resources",
-
+							zap.String("function", fsvc.Function.Name),
-			for _, kubeobj := range fsvc.KubernetesObjects {
+							zap.String("address", fsvc.Address),
-				reaper.CleanupKubeObject(gpm.logger, gpm.kubernetesClient, &kubeobj)
+							zap.String("executor", string(fsvc.Executor)),
-			}
+							zap.String("pod", fsvc.Name),
+						)
+						reaper.CleanupKubeObject(gpm.logger, gpm.kubernetesClient, &fsvc.KubernetesObjects[i])
+						time.Sleep(50 * time.Millisecond)
+					}
+				}
+			}()
 		}
 	}
 }

pkg/executor/fscache/functionServiceCache.go

@@ -33,7 +33,8 @@ import (
 )
 
 type fscRequestType int
-type executorType int
+
+//type executorType int
 
 const (
 	TOUCH fscRequestType = iota
@@ -41,11 +42,6 @@ const (
 	LOG
 )
 
-const (
-	POOLMGR executorType = iota
-	NEWDEPLOY
-)
-
 type (
 	FuncSvc struct {
 		Name              string                  // Name of object
@@ -53,7 +49,7 @@ type (
 		Environment       *fv1.Environment        // function's environment
 		Address           string                  // Host:Port or IP:Port that the function's service can be reached at.
 		KubernetesObjects []apiv1.ObjectReference // Kubernetes Objects (within the function namespace)
-		Executor          executorType
+		Executor          fv1.ExecutorType
 
 		Ctime time.Time
 		Atime time.Time

pkg/executor/reaper/reaper.go

@@ -20,14 +20,15 @@ import (
 	"strings"
 	"time"
 
-	"github.com/fission/fission/pkg/utils"
 	"go.uber.org/zap"
 	apiv1 "k8s.io/api/core/v1"
 	meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/client-go/kubernetes"
 
+	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
 	"github.com/fission/fission/pkg/crd"
 	"github.com/fission/fission/pkg/types"
+	"github.com/fission/fission/pkg/utils"
 )
 
 var (
@@ -35,52 +36,6 @@ var (
 	delOpt            = meta_v1.DeleteOptions{PropagationPolicy: &deletePropagation}
 )
 
-// CleanupOldExecutorObjects cleans up resources created by old executor instances
-func CleanupOldExecutorObjects(logger *zap.Logger, kubernetesClient *kubernetes.Clientset, instanceId string) {
-	go func() {
-		err := cleanup(logger, kubernetesClient, instanceId)
-		if err != nil {
-			// TODO retry reaper; logged and ignored for now
-			logger.Error("Failed to cleanup old executor objects", zap.Error(err))
-		}
-	}()
-}
-
-func cleanup(logger *zap.Logger, client *kubernetes.Clientset, instanceId string) error {
-
-	err := cleanupServices(logger, client, instanceId)
-	if err != nil {
-		return err
-	}
-
-	err = cleanupHpa(logger, client, instanceId)
-	if err != nil {
-		return err
-	}
-
-	// Deployments are used for idle pools and can be cleaned up
-	// immediately.  (We should "adopt" these instead of creating
-	// a new pool.)
-	err = cleanupDeployments(logger, client, instanceId)
-	if err != nil {
-		return err
-	}
-
-	// Pods might still be running user functions, so we give them
-	// a few minutes before terminating them.  This time is the
-	// maximum function runtime, plus the time a router might
-	// still route to an old instance, i.e. router cache expiry
-	// time.
-	time.Sleep(6 * time.Minute)
-
-	err = cleanupPods(logger, client, instanceId)
-	if err != nil {
-		return err
-	}
-
-	return nil
-}
-
 // CleanupKubeObject deletes given kubernetes object
 func CleanupKubeObject(logger *zap.Logger, kubeClient *kubernetes.Clientset, kubeobj *apiv1.ObjectReference) {
 	switch strings.ToLower(kubeobj.Kind) {
@@ -114,28 +69,19 @@ func CleanupKubeObject(logger *zap.Logger, kubeClient *kubernetes.Clientset, kub
 	}
 }
 
-func cleanupDeployments(logger *zap.Logger, client *kubernetes.Clientset, instanceId string) error {
+func CleanupDeployments(logger *zap.Logger, client *kubernetes.Clientset, instanceId string, listOps meta_v1.ListOptions) error {
-	deploymentList, err := client.AppsV1().Deployments(meta_v1.NamespaceAll).List(meta_v1.ListOptions{})
+	deploymentList, err := client.AppsV1().Deployments(meta_v1.NamespaceAll).List(listOps)
 	if err != nil {
 		return err
 	}
 	for _, dep := range deploymentList.Items {
-		id, ok := dep.ObjectMeta.Labels[types.EXECUTOR_INSTANCEID_LABEL]
+		id, ok := dep.ObjectMeta.Annotations[types.EXECUTOR_INSTANCEID_LABEL]
-		if ok && id != instanceId {
+		if !ok {
-			logger.Debug("cleaning up deployment", zap.String("deployment", dep.ObjectMeta.Name))
+			// Backward compatibility with older label name
-			err := client.AppsV1().Deployments(dep.ObjectMeta.Namespace).Delete(dep.ObjectMeta.Name, &delOpt)
+			id, ok = dep.ObjectMeta.Labels[types.EXECUTOR_INSTANCEID_LABEL]
-			if err != nil {
-				logger.Error("error cleaning up deployment",
-					zap.Error(err),
-					zap.String("deployment_name", dep.ObjectMeta.Name),
-					zap.String("deployment_namespace", dep.ObjectMeta.Namespace))
-			}
-			// ignore err
 		}
-		// Backward compatibility with older label name
+		if ok && id != instanceId {
-		pid, pok := dep.ObjectMeta.Labels[types.POOLMGR_INSTANCEID_LABEL]
+			logger.Info("cleaning up deployment", zap.String("deployment", dep.ObjectMeta.Name))
-		if pok && pid != instanceId {
-			logger.Debug("cleaning up deployment", zap.String("deployment", dep.ObjectMeta.Name))
 			err := client.AppsV1().Deployments(dep.ObjectMeta.Namespace).Delete(dep.ObjectMeta.Name, &delOpt)
 			if err != nil {
 				logger.Error("error cleaning up deployment",
@@ -149,15 +95,19 @@ func cleanupDeployments(logger *zap.Logger, client *kubernetes.Clientset, instan
 	return nil
 }
 
-func cleanupPods(logger *zap.Logger, client *kubernetes.Clientset, instanceId string) error {
+func CleanupPods(logger *zap.Logger, client *kubernetes.Clientset, instanceId string, listOps meta_v1.ListOptions) error {
-	podList, err := client.CoreV1().Pods(meta_v1.NamespaceAll).List(meta_v1.ListOptions{})
+	podList, err := client.CoreV1().Pods(meta_v1.NamespaceAll).List(listOps)
 	if err != nil {
 		return err
 	}
 	for _, pod := range podList.Items {
-		id, ok := pod.ObjectMeta.Labels[types.EXECUTOR_INSTANCEID_LABEL]
+		id, ok := pod.ObjectMeta.Annotations[types.EXECUTOR_INSTANCEID_LABEL]
+		if !ok {
+			// Backward compatibility with older label name
+			id, ok = pod.ObjectMeta.Labels[types.EXECUTOR_INSTANCEID_LABEL]
+		}
 		if ok && id != instanceId {
-			logger.Debug("cleaning up pod", zap.String("pod", pod.ObjectMeta.Name))
+			logger.Info("cleaning up pod", zap.String("pod", pod.ObjectMeta.Name))
 			err := client.CoreV1().Pods(pod.ObjectMeta.Namespace).Delete(pod.ObjectMeta.Name, nil)
 			if err != nil {
 				logger.Error("error cleaning up pod",
@@ -167,31 +117,23 @@ func cleanupPods(logger *zap.Logger, client *kubernetes.Clientset, instanceId st
 			}
 			// ignore err
 		}
-		// Backward compatibility with older label name
-		pid, pok := pod.ObjectMeta.Labels[types.POOLMGR_INSTANCEID_LABEL]
-		if pok && pid != instanceId {
-			logger.Debug("cleaning up pod", zap.String("pod", pod.ObjectMeta.Name))
-			err := client.CoreV1().Pods(pod.ObjectMeta.Namespace).Delete(pod.ObjectMeta.Name, nil)
-			if err != nil {
-				logger.Error("error cleaning up pod",
-					zap.Error(err),
-					zap.String("pod_name", pod.ObjectMeta.Name),
-					zap.String("pod_namespace", pod.ObjectMeta.Namespace))
-			}
-		}
 	}
 	return nil
 }
 
-func cleanupServices(logger *zap.Logger, client *kubernetes.Clientset, instanceId string) error {
+func CleanupServices(logger *zap.Logger, client *kubernetes.Clientset, instanceId string, listOps meta_v1.ListOptions) error {
-	svcList, err := client.CoreV1().Services(meta_v1.NamespaceAll).List(meta_v1.ListOptions{})
+	svcList, err := client.CoreV1().Services(meta_v1.NamespaceAll).List(listOps)
 	if err != nil {
 		return err
 	}
 	for _, svc := range svcList.Items {
-		id, ok := svc.ObjectMeta.Labels[types.EXECUTOR_INSTANCEID_LABEL]
+		id, ok := svc.ObjectMeta.Annotations[types.EXECUTOR_INSTANCEID_LABEL]
+		if !ok {
+			// Backward compatibility with older label name
+			id, ok = svc.ObjectMeta.Labels[types.EXECUTOR_INSTANCEID_LABEL]
+		}
 		if ok && id != instanceId {
-			logger.Debug("cleaning up service", zap.String("service", svc.ObjectMeta.Name))
+			logger.Info("cleaning up service", zap.String("service", svc.ObjectMeta.Name))
 			err := client.CoreV1().Services(svc.ObjectMeta.Namespace).Delete(svc.ObjectMeta.Name, nil)
 			if err != nil {
 				logger.Error("error cleaning up service",
@@ -205,16 +147,20 @@ func cleanupServices(logger *zap.Logger, client *kubernetes.Clientset, instanceI
 	return nil
 }
 
-func cleanupHpa(logger *zap.Logger, client *kubernetes.Clientset, instanceId string) error {
+func CleanupHpa(logger *zap.Logger, client *kubernetes.Clientset, instanceId string, listOps meta_v1.ListOptions) error {
-	hpaList, err := client.AutoscalingV1().HorizontalPodAutoscalers(meta_v1.NamespaceAll).List(meta_v1.ListOptions{})
+	hpaList, err := client.AutoscalingV1().HorizontalPodAutoscalers(meta_v1.NamespaceAll).List(listOps)
 	if err != nil {
 		return err
 	}
 
 	for _, hpa := range hpaList.Items {
-		id, ok := hpa.ObjectMeta.Labels[types.EXECUTOR_INSTANCEID_LABEL]
+		id, ok := hpa.ObjectMeta.Annotations[types.EXECUTOR_INSTANCEID_LABEL]
+		if !ok {
+			// Backward compatibility with older label name
+			id, ok = hpa.ObjectMeta.Labels[types.EXECUTOR_INSTANCEID_LABEL]
+		}
 		if ok && id != instanceId {
-			logger.Debug("cleaning up HPA", zap.String("hpa", hpa.ObjectMeta.Name))
+			logger.Info("cleaning up HPA", zap.String("hpa", hpa.ObjectMeta.Name))
 			err := client.AutoscalingV1().HorizontalPodAutoscalers(hpa.ObjectMeta.Namespace).Delete(hpa.ObjectMeta.Name, nil)
 			if err != nil {
 				logger.Error("error cleaning up HPA",
@@ -224,7 +170,6 @@ func cleanupHpa(logger *zap.Logger, client *kubernetes.Clientset, instanceId str
 			}
 			// ignore err
 		}
-
 	}
 	return nil
 
@@ -234,6 +179,9 @@ func cleanupHpa(logger *zap.Logger, client *kubernetes.Clientset, instanceId str
 // deletes the rolebindings completely if there are no Service Accounts in a rolebinding object.
 func CleanupRoleBindings(logger *zap.Logger, client *kubernetes.Clientset, fissionClient *crd.FissionClient, functionNs, envBuilderNs string, cleanupRoleBindingInterval time.Duration) {
 	for {
+		// some sleep before the next reaper iteration
+		time.Sleep(cleanupRoleBindingInterval)
+
 		logger.Debug("starting cleanupRoleBindings cycle")
 		// get all rolebindings ( just to be efficient, one call to kubernetes )
 		rbList, err := client.RbacV1beta1().RoleBindings(meta_v1.NamespaceAll).List(meta_v1.ListOptions{})
@@ -294,7 +242,7 @@ func CleanupRoleBindings(logger *zap.Logger, client *kubernetes.Clientset, fissi
 						break
 					}
 
-					if fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType == types.ExecutorTypeNewdeploy {
+					if fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType == fv1.ExecutorTypeNewdeploy {
 						ndmFunc = true
 						break
 					}
@@ -362,8 +310,5 @@ func CleanupRoleBindings(logger *zap.Logger, client *kubernetes.Clientset, fissi
 				}
 			}
 		}
-
-		// some sleep before the next reaper iteration
-		time.Sleep(cleanupRoleBindingInterval)
 	}
 }

pkg/executor/util/util.go

@@ -0,0 +1,50 @@
+/*
+Copyright 2019 The Fission Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package util
+
+import (
+	"sync"
+	"time"
+
+	apiv1 "k8s.io/api/core/v1"
+)
+
+// ApplyImagePullSecret applies image pull secret to the give pod spec.
+// It's intentional not to check the existence of secret here.
+// First, Kubernetes will set Pod status to "ImagePullBackOff" once
+// kubelet failed to pull image so that users will know what's happening.
+// Second, Fission no longer need to handle "secret not found" error
+// when creating the environment deployment since kubelet will retry to
+// pull image until successes.
+func ApplyImagePullSecret(secret string, podspec apiv1.PodSpec) *apiv1.PodSpec {
+	if len(secret) > 0 {
+		podspec.ImagePullSecrets = []apiv1.LocalObjectReference{{Name: secret}}
+	}
+	return &podspec
+}
+
+func WaitTimeout(wg *sync.WaitGroup, timeout time.Duration) {
+	waitCh := make(chan struct{})
+	go func() {
+		defer close(waitCh)
+		wg.Wait()
+	}()
+	select {
+	case <-waitCh:
+	case <-time.After(timeout):
+	}
+}

pkg/fetcher/client/client.go

@@ -9,9 +9,9 @@ import (
 	"strings"
 	"time"
 
-	"go.uber.org/zap"
+	"github.com/pkg/errors"
-
 	"go.opencensus.io/plugin/ochttp"
+	"go.uber.org/zap"
 	"golang.org/x/net/context/ctxhttp"
 
 	ferror "github.com/fission/fission/pkg/error"
@@ -91,12 +91,20 @@ func sendRequest(logger *zap.Logger, ctx context.Context, httpClient *http.Clien
 				if err != nil {
 					logger.Error("error reading response body", zap.Error(err))
 				}
-				resp.Body.Close()
+				defer resp.Body.Close()
 				return body, err
 			}
 			err = ferror.MakeErrorFromHTTP(resp)
 		}
 
+		// skip retry and return directly due to context deadline exceeded
+		if err == context.DeadlineExceeded {
+			msg := "error specializing function pod, either increase the specialization timeout for function or check function pod log would help."
+			err = errors.Wrap(err, msg)
+			logger.Error(msg, zap.Error(err), zap.String("url", url))
+			return nil, err
+		}
+
 		if i < maxRetries-1 {
 			time.Sleep(50 * time.Duration(2*i) * time.Millisecond)
 			logger.Error("error specializing/fetching/uploading package, retrying", zap.Error(err), zap.String("url", url))

pkg/fetcher/fetcher.go

@@ -21,7 +21,6 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
-	"io"
 	"io/ioutil"
 	"net/http"
 	"os"
@@ -33,7 +32,6 @@ import (
 	uuid "github.com/satori/go.uuid"
 	"go.opencensus.io/plugin/ochttp"
 	"go.uber.org/zap"
-	"golang.org/x/net/context/ctxhttp"
 	k8serr "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/client-go/kubernetes"
@@ -96,38 +94,6 @@ func MakeFetcher(logger *zap.Logger, sharedVolumePath string, sharedSecretPath s
 	}, nil
 }
 
-func downloadUrl(ctx context.Context, httpClient *http.Client, url string, localPath string) error {
-	resp, err := ctxhttp.Get(ctx, httpClient, url)
-	if err != nil {
-		return err
-	}
-	defer resp.Body.Close()
-
-	w, err := os.Create(localPath)
-	if err != nil {
-		return err
-	}
-	defer w.Close()
-
-	_, err = io.Copy(w, resp.Body)
-	if err != nil {
-		return err
-	}
-
-	// flushing write buffer to file
-	err = w.Sync()
-	if err != nil {
-		return err
-	}
-
-	err = os.Chmod(localPath, 0600)
-	if err != nil {
-		return err
-	}
-
-	return nil
-}
-
 func verifyChecksum(fileChecksum, checksum *fv1.Checksum) error {
 	if checksum.Type != fv1.ChecksumTypeSHA256 {
 		return ferror.MakeError(ferror.ErrorInvalidArgument, "Unsupported checksum type")
@@ -232,7 +198,7 @@ func (fetcher *Fetcher) SpecializeHandler(w http.ResponseWriter, r *http.Request
 	err = fetcher.SpecializePod(r.Context(), req.FetchReq, req.LoadReq)
 	if err != nil {
 		fetcher.logger.Error("error specializing pod", zap.Error(err))
-		w.WriteHeader(http.StatusInternalServerError)
+		http.Error(w, err.Error(), http.StatusInternalServerError)
 		return
 	}
 
@@ -263,7 +229,7 @@ func (fetcher *Fetcher) Fetch(ctx context.Context, pkg *fv1.Package, req types.F
 
 	if req.FetchType == types.FETCH_URL {
 		// fetch the file and save it to the tmp path
-		err := downloadUrl(ctx, fetcher.httpClient, req.Url, tmpPath)
+		err := utils.DownloadUrl(ctx, fetcher.httpClient, req.Url, tmpPath)
 		if err != nil {
 			e := "failed to download url"
 			fetcher.logger.Error(e, zap.Error(err), zap.String("url", req.Url))
@@ -302,7 +268,7 @@ func (fetcher *Fetcher) Fetch(ctx context.Context, pkg *fv1.Package, req types.F
 			}
 		} else {
 			// download and verify
-			err := downloadUrl(ctx, fetcher.httpClient, archive.URL, tmpPath)
+			err := utils.DownloadUrl(ctx, fetcher.httpClient, archive.URL, tmpPath)
 			if err != nil {
 				e := "failed to download url"
 				fetcher.logger.Error(e, zap.Error(err), zap.String("url", req.Url))

pkg/fission-cli/cliwrapper/driver/cobra/cobra.go

@@ -53,6 +53,14 @@ func Wrapper(action cmd.CommandAction) func(*cobra.Command, []string) error {
 func SetFlags(cmd *cobra.Command, flagSet flag.FlagSet) {
 	aliases := make(map[string]string)
 
+	// set global flags
+	for _, f := range flagSet.Global {
+		globalFlags(cmd, f)
+		for _, alias := range f.Aliases {
+			aliases[alias] = f.Name
+		}
+	}
+
 	// set required flags
 	for _, f := range flagSet.Required {
 		requiredFlags(cmd, f)
@@ -84,7 +92,7 @@ func SetFlags(cmd *cobra.Command, flagSet flag.FlagSet) {
 
 func optionalFlags(cmd *cobra.Command, flags ...flag.Flag) {
 	for _, f := range flags {
-		toCobraFlag(cmd, f)
+		toCobraFlag(cmd, f, false)
 		if f.Deprecated {
 			usage := fmt.Sprintf("Use --%v instead. The flag still works for now and will be removed in future", f.Substitute)
 			cmd.Flags().MarkDeprecated(f.Name, usage)
@@ -96,12 +104,18 @@ func optionalFlags(cmd *cobra.Command, flags ...flag.Flag) {
 
 func requiredFlags(cmd *cobra.Command, flags ...flag.Flag) {
 	for _, f := range flags {
-		toCobraFlag(cmd, f)
+		toCobraFlag(cmd, f, false)
 		cmd.MarkFlagRequired(f.Name)
 	}
 }
 
-func toCobraFlag(cmd *cobra.Command, f flag.Flag) {
+func globalFlags(cmd *cobra.Command, flags ...flag.Flag) {
+	for _, f := range flags {
+		toCobraFlag(cmd, f, true)
+	}
+}
+
+func toCobraFlag(cmd *cobra.Command, f flag.Flag, global bool) {
 	// Workaround to pass aliases to templater for generating flag aliases.
 	if len(f.Aliases) > 0 {
 		var aliases []string
@@ -119,67 +133,72 @@ func toCobraFlag(cmd *cobra.Command, f flag.Flag) {
 			helptemplate.AliasSeparator, f.Usage)
 	}
 
+	flagset := cmd.Flags()
+	if global {
+		flagset = cmd.PersistentFlags()
+	}
+
 	switch f.Type {
 	case flag.Bool:
 		val, ok := f.DefaultValue.(bool)
 		if !ok {
 			val = false
 		}
-		cmd.Flags().BoolP(f.Name, f.Short, val, f.Usage)
+		flagset.BoolP(f.Name, f.Short, val, f.Usage)
 	case flag.String:
 		val, ok := f.DefaultValue.(string)
 		if !ok {
 			val = ""
 		}
-		cmd.Flags().StringP(f.Name, f.Short, val, f.Usage)
+		flagset.StringP(f.Name, f.Short, val, f.Usage)
 	case flag.StringSlice:
 		val, ok := f.DefaultValue.([]string)
 		if !ok {
 			val = []string{}
 		}
-		cmd.Flags().StringArrayP(f.Name, f.Short, val, f.Usage)
+		flagset.StringArrayP(f.Name, f.Short, val, f.Usage)
 	case flag.Int:
 		val, ok := f.DefaultValue.(int)
 		if !ok {
 			val = 0
 		}
-		cmd.Flags().IntP(f.Name, f.Short, val, f.Usage)
+		flagset.IntP(f.Name, f.Short, val, f.Usage)
 	case flag.IntSlice:
 		val, ok := f.DefaultValue.([]int)
 		if !ok {
 			val = []int{}
 		}
-		cmd.Flags().IntSliceP(f.Name, f.Short, val, f.Usage)
+		flagset.IntSliceP(f.Name, f.Short, val, f.Usage)
 	case flag.Int64:
 		val, ok := f.DefaultValue.(int64)
 		if !ok {
 			val = 0
 		}
-		cmd.Flags().Int64P(f.Name, f.Short, val, f.Usage)
+		flagset.Int64P(f.Name, f.Short, val, f.Usage)
 	case flag.Int64Slice:
 		val, ok := f.DefaultValue.([]int64)
 		if !ok {
 			val = []int64{}
 		}
-		cmd.Flags().Int64SliceP(f.Name, f.Short, val, f.Usage)
+		flagset.Int64SliceP(f.Name, f.Short, val, f.Usage)
 	case flag.Float32:
 		val, ok := f.DefaultValue.(float32)
 		if !ok {
 			val = 0
 		}
-		cmd.Flags().Float32P(f.Name, f.Short, val, f.Usage)
+		flagset.Float32P(f.Name, f.Short, val, f.Usage)
 	case flag.Float64:
 		val, ok := f.DefaultValue.(float64)
 		if !ok {
 			val = 0
 		}
-		cmd.Flags().Float64P(f.Name, f.Short, val, f.Usage)
+		flagset.Float64P(f.Name, f.Short, val, f.Usage)
 	case flag.Duration:
 		val, ok := f.DefaultValue.(time.Duration)
 		if !ok {
 			val = 0
 		}
-		cmd.Flags().DurationP(f.Name, f.Short, val, f.Usage)
+		flagset.DurationP(f.Name, f.Short, val, f.Usage)
 	}
 }
 

pkg/fission-cli/cliwrapper/driver/cobra/helptemplate/templates.go

@@ -50,7 +50,7 @@ const (
 {{end}}`
 
 	// SectionFlags is the help template section that displays the command's flags.
-	SectionFlags = `{{ if or $visibleFlags.HasFlags $explicitlyExposedFlags.HasFlags}}Options:
+	SectionFlags = `{{ if or $visibleFlags.HasFlags $explicitlyExposedFlags.HasFlags}}Options
 {{ if $visibleFlags.HasFlags}}{{trimRight (flagsUsages $visibleFlags)}}{{end}}{{ if $explicitlyExposedFlags.HasFlags}}{{ if $visibleFlags.HasFlags}}
 {{end}}{{trimRight (flagsUsages $explicitlyExposedFlags)}}{{end}}
 

pkg/fission-cli/cmd/environment/command.go

@@ -33,8 +33,8 @@ func Commands() *cobra.Command {
 		Required: []flag.Flag{flag.EnvName, flag.EnvImage},
 		Optional: []flag.Flag{flag.EnvPoolsize, flag.EnvBuilderImage, flag.EnvBuildCmd,
 			flag.RunTimeMinCPU, flag.RunTimeMaxCPU, flag.RunTimeMinMemory, flag.RunTimeMaxMemory,
-			flag.EnvTerminationGracePeriod, flag.EnvVersion, flag.EnvExternalNetwork, flag.EnvKeepArchive,
+			flag.EnvTerminationGracePeriod, flag.EnvVersion, flag.EnvImagePullSecret,
-			flag.NamespaceEnvironment, flag.SpecSave},
+			flag.EnvExternalNetwork, flag.EnvKeepArchive, flag.NamespaceEnvironment, flag.SpecSave},
 	})
 
 	getCmd := &cobra.Command{
@@ -55,8 +55,8 @@ func Commands() *cobra.Command {
 	wrapper.SetFlags(updateCmd, flag.FlagSet{
 		Required: []flag.Flag{flag.EnvName},
 		Optional: []flag.Flag{flag.EnvImage, flag.EnvPoolsize,
-			flag.EnvBuilderImage, flag.EnvBuildCmd, flag.EnvExternalNetwork,
+			flag.EnvBuilderImage, flag.EnvBuildCmd, flag.EnvImagePullSecret, flag.EnvTerminationGracePeriod,
-			flag.EnvTerminationGracePeriod, flag.EnvKeepArchive, flag.NamespaceEnvironment},
+			flag.EnvKeepArchive, flag.NamespaceEnvironment, flag.EnvExternalNetwork},
 	})
 
 	deleteCmd := &cobra.Command{

pkg/fission-cli/cmd/environment/create.go

@@ -112,6 +112,7 @@ func createEnvironmentFromCmd(input cli.Input) (*fv1.Environment, error) {
 	envExternalNetwork := input.Bool(flagkey.EnvExternalNetwork)
 	keepArchive := input.Bool(flagkey.EnvKeeparchive)
 	envGracePeriod := input.Int64(flagkey.EnvGracePeriod)
+	pullSecret := input.String(flagkey.EnvImagePullSecret)
 
 	envVersion := input.Int(flagkey.EnvVersion)
 	// Environment API interface version is not specified and
@@ -169,6 +170,7 @@ func createEnvironmentFromCmd(input cli.Input) (*fv1.Environment, error) {
 			AllowAccessToExternalNetwork: envExternalNetwork,
 			TerminationGracePeriod:       envGracePeriod,
 			KeepArchive:                  keepArchive,
+			ImagePullSecret:              pullSecret,
 		},
 	}
 

pkg/fission-cli/cmd/environment/update.go

@@ -123,6 +123,10 @@ func updateExistingEnvironmentWithCmd(env *fv1.Environment, input cli.Input) (*f
 		env.Spec.KeepArchive = input.Bool(flagkey.EnvKeeparchive)
 	}
 
+	if input.IsSet(flagkey.EnvImagePullSecret) {
+		env.Spec.ImagePullSecret = input.String(flagkey.EnvImagePullSecret)
+	}
+
 	env.Spec.AllowAccessToExternalNetwork = envExternalNetwork
 
 	// TODO: allow to update resource.

pkg/fission-cli/cmd/function/command.go

@@ -36,13 +36,16 @@ func Commands() *cobra.Command {
 			flag.FnExecutorType, flag.FnCfgMap, flag.FnSecret,
 			flag.FnSpecializationTimeout, flag.FnExecutionTimeout,
 
-			// TODO retired pkg related flag from function cmd
+			// TODO retired pkg & trigger related flags from function cmd
 			flag.PkgCode, flag.PkgSrcArchive, flag.PkgDeployArchive,
+			flag.PkgSrcChecksum, flag.PkgDeployChecksum, flag.PkgInsecure,
+			flag.FnBuildCmd,
+
 			flag.HtUrl, flag.HtMethod,
 
 			// flag for newdeploy to use.
 			flag.RunTimeMinCPU, flag.RunTimeMaxCPU, flag.RunTimeMinMemory,
-			flag.RunTimeMaxMemory, flag.FnBuildCmd, flag.ReplicasMin,
+			flag.RunTimeMaxMemory, flag.ReplicasMin,
 			flag.ReplicasMax, flag.RunTimeTargetCPU,
 
 			flag.NamespaceFunction, flag.NamespaceEnvironment, flag.SpecSave},
@@ -84,6 +87,7 @@ func Commands() *cobra.Command {
 			flag.FnSpecializationTimeout, flag.FnExecutionTimeout,
 
 			flag.PkgCode, flag.PkgSrcArchive, flag.PkgDeployArchive,
+			flag.PkgSrcChecksum, flag.PkgDeployChecksum, flag.PkgInsecure,
 			flag.FnBuildCmd, flag.PkgForce,
 
 			flag.RunTimeMinCPU, flag.RunTimeMaxCPU, flag.RunTimeMinMemory,

pkg/fission-cli/cmd/function/create.go

@@ -36,7 +36,6 @@ import (
 	"github.com/fission/fission/pkg/fission-cli/console"
 	flagkey "github.com/fission/fission/pkg/fission-cli/flag/key"
 	"github.com/fission/fission/pkg/fission-cli/util"
-	"github.com/fission/fission/pkg/types"
 )
 
 const (
@@ -363,36 +362,48 @@ func (opts *CreateSubCommand) run(input cli.Input) error {
 }
 
 func getInvokeStrategy(input cli.Input, existingInvokeStrategy *fv1.InvokeStrategy) (strategy *fv1.InvokeStrategy, err error) {
+	var es *fv1.ExecutionStrategy
 
-	var fnExecutor, newFnExecutor fv1.ExecutorType
+	if existingInvokeStrategy == nil {
+		es, err = getExecutionStrategy(input)
+	} else {
+		es, err = updateExecutionStrategy(input, &existingInvokeStrategy.ExecutionStrategy)
+	}
+
+	if err != nil {
+		return nil, err
+	}
+
+	return &fv1.InvokeStrategy{
+		ExecutionStrategy: *es,
+		StrategyType:      fv1.StrategyTypeExecution,
+	}, nil
+}
+
+func getExecutionStrategy(input cli.Input) (strategy *fv1.ExecutionStrategy, err error) {
+	var fnExecutor fv1.ExecutorType
 
 	switch input.String(flagkey.FnExecutorType) {
 	case "":
 		fallthrough
-	case types.ExecutorTypePoolmgr:
+	case string(fv1.ExecutorTypePoolmgr):
-		newFnExecutor = types.ExecutorTypePoolmgr
+		fnExecutor = fv1.ExecutorTypePoolmgr
-	case types.ExecutorTypeNewdeploy:
+	case string(fv1.ExecutorTypeNewdeploy):
-		newFnExecutor = types.ExecutorTypeNewdeploy
+		fnExecutor = fv1.ExecutorTypeNewdeploy
 	default:
-		return nil, errors.New("executor type must be one of 'poolmgr' or 'newdeploy', defaults to 'poolmgr'")
+		return nil, errors.Errorf("executor type must be one of '%v' or '%v'", fv1.ExecutorTypePoolmgr, fv1.ExecutorTypeNewdeploy)
 	}
 
-	if existingInvokeStrategy != nil {
+	specializationTimeout := fv1.DefaultSpecializationTimeOut
-		fnExecutor = existingInvokeStrategy.ExecutionStrategy.ExecutorType
 
-		// override the executor type if user specified a new executor type
+	if input.IsSet(flagkey.FnSpecializationTimeout) {
-		if input.IsSet(flagkey.FnExecutorType) {
+		specializationTimeout = input.Int(flagkey.FnSpecializationTimeout)
-			fnExecutor = newFnExecutor
+		if specializationTimeout < fv1.DefaultSpecializationTimeOut {
+			return nil, errors.Errorf("%v must be greater than or equal to 120 seconds", flagkey.FnSpecializationTimeout)
 		}
-	} else {
-		fnExecutor = newFnExecutor
 	}
 
-	if input.IsSet(flagkey.FnSpecializationTimeout) && fnExecutor != types.ExecutorTypeNewdeploy {
+	if fnExecutor == fv1.ExecutorTypePoolmgr {
-		return nil, errors.Errorf("%v flag is only applicable for newdeploy type of executor", flagkey.FnSpecializationTimeout)
-	}
-
-	if fnExecutor == types.ExecutorTypePoolmgr {
 		if input.IsSet(flagkey.RuntimeTargetcpu) || input.IsSet(flagkey.ReplicasMinscale) || input.IsSet(flagkey.ReplicasMaxscale) {
 			return nil, errors.New("to set target CPU or min/max scale for function, please specify \"--executortype newdeploy\"")
 		}
@@ -400,24 +411,102 @@ func getInvokeStrategy(input cli.Input, existingInvokeStrategy *fv1.InvokeStrate
 		if input.IsSet(flagkey.RuntimeMincpu) || input.IsSet(flagkey.RuntimeMaxcpu) || input.IsSet(flagkey.RuntimeMinmemory) || input.IsSet(flagkey.RuntimeMaxmemory) {
 			console.Warn("To limit CPU/Memory for function with executor type \"poolmgr\", please specify resources limits when creating environment")
 		}
-		strategy = &fv1.InvokeStrategy{
+
-			StrategyType: fv1.StrategyTypeExecution,
+		strategy = &fv1.ExecutionStrategy{
-			ExecutionStrategy: fv1.ExecutionStrategy{
+			ExecutorType:          fv1.ExecutorTypePoolmgr,
-				ExecutorType: types.ExecutorTypePoolmgr,
+			SpecializationTimeout: specializationTimeout,
-			},
 		}
 	} else {
-		// set default value
 		targetCPU := DEFAULT_TARGET_CPU_PERCENTAGE
-		minScale := DEFAULT_MIN_SCALE
+		if input.IsSet(flagkey.RuntimeTargetcpu) {
-		maxScale := minScale
+			targetCPU, err = getTargetCPU(input)
-		specializationTimeout := fv1.DefaultSpecializationTimeOut
+			if err != nil {
+				return nil, err
+			}
+		}
 
-		if existingInvokeStrategy != nil && existingInvokeStrategy.ExecutionStrategy.ExecutorType == types.ExecutorTypeNewdeploy {
+		minScale := DEFAULT_MIN_SCALE
-			minScale = existingInvokeStrategy.ExecutionStrategy.MinScale
+		if input.IsSet(flagkey.ReplicasMinscale) {
-			maxScale = existingInvokeStrategy.ExecutionStrategy.MaxScale
+			minScale = input.Int(flagkey.ReplicasMinscale)
-			targetCPU = existingInvokeStrategy.ExecutionStrategy.TargetCPUPercent
+		}
-			specializationTimeout = existingInvokeStrategy.ExecutionStrategy.SpecializationTimeout
+
+		maxScale := minScale
+		if input.IsSet(flagkey.ReplicasMaxscale) {
+			maxScale = input.Int(flagkey.ReplicasMaxscale)
+			if maxScale <= 0 {
+				return nil, errors.Errorf("%v must be greater than 0", flagkey.ReplicasMaxscale)
+			}
+		}
+
+		if minScale > maxScale {
+			return nil, fmt.Errorf("minscale (%v) can not be greater than maxscale (%v)", minScale, maxScale)
+		}
+
+		// Right now a simple single case strategy implementation
+		// This will potentially get more sophisticated once we have more strategies in place
+		strategy = &fv1.ExecutionStrategy{
+			ExecutorType:          fnExecutor,
+			MinScale:              minScale,
+			MaxScale:              maxScale,
+			TargetCPUPercent:      targetCPU,
+			SpecializationTimeout: specializationTimeout,
+		}
+	}
+
+	return strategy, nil
+}
+
+func updateExecutionStrategy(input cli.Input, existingExecutionStrategy *fv1.ExecutionStrategy) (strategy *fv1.ExecutionStrategy, err error) {
+	fnExecutor := existingExecutionStrategy.ExecutorType
+	oldExecutor := existingExecutionStrategy.ExecutorType
+
+	if input.IsSet(flagkey.FnExecutorType) {
+		switch input.String(flagkey.FnExecutorType) {
+		case "":
+			fallthrough
+		case string(fv1.ExecutorTypePoolmgr):
+			fnExecutor = fv1.ExecutorTypePoolmgr
+		case string(fv1.ExecutorTypeNewdeploy):
+			fnExecutor = fv1.ExecutorTypeNewdeploy
+		default:
+			return nil, errors.Errorf("executor type must be one of '%v' or '%v'", fv1.ExecutorTypePoolmgr, fv1.ExecutorTypeNewdeploy)
+		}
+	}
+
+	specializationTimeout := existingExecutionStrategy.SpecializationTimeout
+
+	if input.IsSet(flagkey.FnSpecializationTimeout) {
+		specializationTimeout = input.Int(flagkey.FnSpecializationTimeout)
+		if specializationTimeout < fv1.DefaultSpecializationTimeOut {
+			return nil, errors.Errorf("%v must be greater than or equal to 120 seconds", flagkey.FnSpecializationTimeout)
+		}
+	} else {
+		if specializationTimeout < fv1.DefaultSpecializationTimeOut {
+			specializationTimeout = fv1.DefaultSpecializationTimeOut
+		}
+	}
+
+	if fnExecutor == fv1.ExecutorTypePoolmgr {
+		if input.IsSet(flagkey.RuntimeTargetcpu) || input.IsSet(flagkey.ReplicasMinscale) || input.IsSet(flagkey.ReplicasMaxscale) {
+			return nil, errors.New("to set target CPU or min/max scale for function, please specify \"--executortype newdeploy\"")
+		}
+
+		if input.IsSet(flagkey.RuntimeMincpu) || input.IsSet(flagkey.RuntimeMaxcpu) || input.IsSet(flagkey.RuntimeMinmemory) || input.IsSet(flagkey.RuntimeMaxmemory) {
+			console.Warn("To limit CPU/Memory for function with executor type \"poolmgr\", please specify resources limits when creating environment")
+		}
+		strategy = &fv1.ExecutionStrategy{
+			ExecutorType:          fv1.ExecutorTypePoolmgr,
+			SpecializationTimeout: specializationTimeout,
+		}
+	} else {
+		targetCPU := existingExecutionStrategy.TargetCPUPercent
+		minScale := existingExecutionStrategy.MinScale
+		maxScale := existingExecutionStrategy.MaxScale
+
+		if fnExecutor != oldExecutor { // from poolmanager to newdeploy
+			targetCPU = DEFAULT_TARGET_CPU_PERCENTAGE
+			minScale = DEFAULT_MIN_SCALE
+			maxScale = minScale
 		}
 
 		if input.IsSet(flagkey.RuntimeTargetcpu) {
@@ -425,6 +514,10 @@ func getInvokeStrategy(input cli.Input, existingInvokeStrategy *fv1.InvokeStrate
 			if err != nil {
 				return nil, err
 			}
+		} else {
+			if targetCPU <= 0 || targetCPU > 100 {
+				targetCPU = DEFAULT_TARGET_CPU_PERCENTAGE
+			}
 		}
 
 		if input.IsSet(flagkey.ReplicasMinscale) {
@@ -436,12 +529,9 @@ func getInvokeStrategy(input cli.Input, existingInvokeStrategy *fv1.InvokeStrate
 			if maxScale <= 0 {
 				return nil, errors.Errorf("%v must be greater than 0", flagkey.ReplicasMaxscale)
 			}
-		}
+		} else {
-
+			if maxScale <= 0 {
-		if input.IsSet(flagkey.FnSpecializationTimeout) {
+				maxScale = 1
-			specializationTimeout = input.Int(flagkey.FnSpecializationTimeout)
-			if specializationTimeout < fv1.DefaultSpecializationTimeOut {
-				return nil, errors.Errorf("%v must be greater than or equal to 120 seconds", flagkey.FnSpecializationTimeout)
 			}
 		}
 
@@ -451,15 +541,12 @@ func getInvokeStrategy(input cli.Input, existingInvokeStrategy *fv1.InvokeStrate
 
 		// Right now a simple single case strategy implementation
 		// This will potentially get more sophisticated once we have more strategies in place
-		strategy = &fv1.InvokeStrategy{
+		strategy = &fv1.ExecutionStrategy{
-			StrategyType: fv1.StrategyTypeExecution,
+			ExecutorType:          fnExecutor,
-			ExecutionStrategy: fv1.ExecutionStrategy{
+			MinScale:              minScale,
-				ExecutorType:          fnExecutor,
+			MaxScale:              maxScale,
-				MinScale:              minScale,
+			TargetCPUPercent:      targetCPU,
-				MaxScale:              maxScale,
+			SpecializationTimeout: specializationTimeout,
-				TargetCPUPercent:      targetCPU,
-				SpecializationTimeout: specializationTimeout,
-			},
 		}
 	}
 

pkg/fission-cli/cmd/function/function_test.go

@@ -29,38 +29,41 @@ import (
 
 func TestGetInvokeStrategy(t *testing.T) {
 	cases := []struct {
+		name                   string
 		testArgs               map[string]interface{}
 		existingInvokeStrategy *fv1.InvokeStrategy
 		expectedResult         *fv1.InvokeStrategy
 		expectError            bool
 	}{
 		{
-			// case: use default executor poolmgr
+			name:                   "use default executor poolmgr",
 			testArgs:               map[string]interface{}{},
 			existingInvokeStrategy: nil,
 			expectedResult: &fv1.InvokeStrategy{
 				StrategyType: fv1.StrategyTypeExecution,
 				ExecutionStrategy: fv1.ExecutionStrategy{
-					ExecutorType: fv1.ExecutorTypePoolmgr,
+					ExecutorType:          fv1.ExecutorTypePoolmgr,
+					SpecializationTimeout: 120,
 				},
 			},
 			expectError: false,
 		},
 		{
-			// case: executor type set to poolmgr
+			name:                   "executor type set to poolmgr",
-			testArgs:               map[string]interface{}{flagkey.FnExecutorType: fv1.ExecutorTypePoolmgr},
+			testArgs:               map[string]interface{}{flagkey.FnExecutorType: string(fv1.ExecutorTypePoolmgr)},
 			existingInvokeStrategy: nil,
 			expectedResult: &fv1.InvokeStrategy{
 				StrategyType: fv1.StrategyTypeExecution,
 				ExecutionStrategy: fv1.ExecutionStrategy{
-					ExecutorType: fv1.ExecutorTypePoolmgr,
+					ExecutorType:          fv1.ExecutorTypePoolmgr,
+					SpecializationTimeout: 120,
 				},
 			},
 			expectError: false,
 		},
 		{
-			// case: executor type set to newdeploy
+			name:                   "executor type set to newdeploy",
-			testArgs:               map[string]interface{}{flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy},
+			testArgs:               map[string]interface{}{flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy)},
 			existingInvokeStrategy: nil,
 			expectedResult: &fv1.InvokeStrategy{
 				StrategyType: fv1.StrategyTypeExecution,
@@ -75,8 +78,8 @@ func TestGetInvokeStrategy(t *testing.T) {
 			expectError: false,
 		},
 		{
-			// case: executor type change from poolmgr to newdeploy
+			name:     "executor type change from poolmgr to newdeploy",
-			testArgs: map[string]interface{}{flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy},
+			testArgs: map[string]interface{}{flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy)},
 			existingInvokeStrategy: &fv1.InvokeStrategy{
 				StrategyType: fv1.StrategyTypeExecution,
 				ExecutionStrategy: fv1.ExecutionStrategy{
@@ -96,8 +99,8 @@ func TestGetInvokeStrategy(t *testing.T) {
 			expectError: false,
 		},
 		{
-			// case: executor type change from newdeploy to poolmgr
+			name:     "executor type change from newdeploy to poolmgr",
-			testArgs: map[string]interface{}{flagkey.FnExecutorType: fv1.ExecutorTypePoolmgr},
+			testArgs: map[string]interface{}{flagkey.FnExecutorType: string(fv1.ExecutorTypePoolmgr)},
 			existingInvokeStrategy: &fv1.InvokeStrategy{
 				StrategyType: fv1.StrategyTypeExecution,
 				ExecutionStrategy: fv1.ExecutionStrategy{
@@ -111,15 +114,16 @@ func TestGetInvokeStrategy(t *testing.T) {
 			expectedResult: &fv1.InvokeStrategy{
 				StrategyType: fv1.StrategyTypeExecution,
 				ExecutionStrategy: fv1.ExecutionStrategy{
-					ExecutorType: fv1.ExecutorTypePoolmgr,
+					ExecutorType:          fv1.ExecutorTypePoolmgr,
+					SpecializationTimeout: fv1.DefaultSpecializationTimeOut,
 				},
 			},
 			expectError: false,
 		},
 		{
-			// case: minscale < maxscale
+			name: "minscale < maxscale",
 			testArgs: map[string]interface{}{
-				flagkey.FnExecutorType:   fv1.ExecutorTypeNewdeploy,
+				flagkey.FnExecutorType:   string(fv1.ExecutorTypeNewdeploy),
 				flagkey.ReplicasMinscale: 2,
 				flagkey.ReplicasMaxscale: 3,
 			},
@@ -137,9 +141,9 @@ func TestGetInvokeStrategy(t *testing.T) {
 			expectError: false,
 		},
 		{
-			// case: minscale > maxscale
+			name: "minscale > maxscale",
 			testArgs: map[string]interface{}{
-				flagkey.FnExecutorType:   fv1.ExecutorTypeNewdeploy,
+				flagkey.FnExecutorType:   string(fv1.ExecutorTypeNewdeploy),
 				flagkey.ReplicasMinscale: 5,
 				flagkey.ReplicasMaxscale: 3,
 			},
@@ -148,19 +152,28 @@ func TestGetInvokeStrategy(t *testing.T) {
 			expectError:            true,
 		},
 		{
-			// case: maxscale not specified
+			name: "maxscale not specified",
 			testArgs: map[string]interface{}{
-				flagkey.FnExecutorType:   fv1.ExecutorTypeNewdeploy,
+				flagkey.FnExecutorType:   string(fv1.ExecutorTypeNewdeploy),
 				flagkey.ReplicasMinscale: 5,
 			},
 			existingInvokeStrategy: nil,
-			expectedResult:         nil,
+			expectedResult: &fv1.InvokeStrategy{
-			expectError:            true,
+				StrategyType: fv1.StrategyTypeExecution,
+				ExecutionStrategy: fv1.ExecutionStrategy{
+					ExecutorType:          fv1.ExecutorTypeNewdeploy,
+					MinScale:              5,
+					MaxScale:              5,
+					TargetCPUPercent:      DEFAULT_TARGET_CPU_PERCENTAGE,
+					SpecializationTimeout: fv1.DefaultSpecializationTimeOut,
+				},
+			},
+			expectError: false,
 		},
 		{
-			// case: minscale not specified
+			name: "minscale not specified",
 			testArgs: map[string]interface{}{
-				flagkey.FnExecutorType:   fv1.ExecutorTypeNewdeploy,
+				flagkey.FnExecutorType:   string(fv1.ExecutorTypeNewdeploy),
 				flagkey.ReplicasMaxscale: 3,
 			},
 			existingInvokeStrategy: nil,
@@ -177,9 +190,9 @@ func TestGetInvokeStrategy(t *testing.T) {
 			expectError: false,
 		},
 		{
-			// case: maxscale set to 0
+			name: "maxscale set to 0",
 			testArgs: map[string]interface{}{
-				flagkey.FnExecutorType:   fv1.ExecutorTypeNewdeploy,
+				flagkey.FnExecutorType:   string(fv1.ExecutorTypeNewdeploy),
 				flagkey.ReplicasMaxscale: 0,
 			},
 			existingInvokeStrategy: nil,
@@ -187,9 +200,28 @@ func TestGetInvokeStrategy(t *testing.T) {
 			expectError:            true,
 		},
 		{
-			// case: maxscale set to 9 when existing is 5
+			name: "update minscale with value larger than existing maxScale",
 			testArgs: map[string]interface{}{
-				flagkey.FnExecutorType:   fv1.ExecutorTypeNewdeploy,
+				flagkey.FnExecutorType:   string(fv1.ExecutorTypeNewdeploy),
+				flagkey.ReplicasMinscale: 9,
+			},
+			existingInvokeStrategy: &fv1.InvokeStrategy{
+				StrategyType: fv1.StrategyTypeExecution,
+				ExecutionStrategy: fv1.ExecutionStrategy{
+					ExecutorType:          fv1.ExecutorTypeNewdeploy,
+					MinScale:              2,
+					MaxScale:              5,
+					TargetCPUPercent:      DEFAULT_TARGET_CPU_PERCENTAGE,
+					SpecializationTimeout: fv1.DefaultSpecializationTimeOut,
+				},
+			},
+			expectedResult: nil,
+			expectError:    true,
+		},
+		{
+			name: "maxscale set to 9 when existing is 5",
+			testArgs: map[string]interface{}{
+				flagkey.FnExecutorType:   string(fv1.ExecutorTypeNewdeploy),
 				flagkey.ReplicasMaxscale: 9,
 			},
 			existingInvokeStrategy: &fv1.InvokeStrategy{
@@ -215,9 +247,9 @@ func TestGetInvokeStrategy(t *testing.T) {
 			expectError: false,
 		},
 		{
-			// case: change nothing for existing strategy
+			name: "change nothing for existing strategy",
 			testArgs: map[string]interface{}{
-				flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy,
+				flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy),
 			},
 			existingInvokeStrategy: &fv1.InvokeStrategy{
 				StrategyType: fv1.StrategyTypeExecution,
@@ -242,9 +274,9 @@ func TestGetInvokeStrategy(t *testing.T) {
 			expectError: false,
 		},
 		{
-			// case: set target cpu percentage
+			name: "set target cpu percentage",
 			testArgs: map[string]interface{}{
-				flagkey.FnExecutorType:   fv1.ExecutorTypeNewdeploy,
+				flagkey.FnExecutorType:   string(fv1.ExecutorTypeNewdeploy),
 				flagkey.RuntimeTargetcpu: 50,
 			},
 			existingInvokeStrategy: nil,
@@ -261,9 +293,9 @@ func TestGetInvokeStrategy(t *testing.T) {
 			expectError: false,
 		},
 		{
-			// case: change target cpu percentage
+			name: "change target cpu percentage",
 			testArgs: map[string]interface{}{
-				flagkey.FnExecutorType:   fv1.ExecutorTypeNewdeploy,
+				flagkey.FnExecutorType:   string(fv1.ExecutorTypeNewdeploy),
 				flagkey.RuntimeTargetcpu: 20,
 			},
 			existingInvokeStrategy: &fv1.InvokeStrategy{
@@ -289,9 +321,9 @@ func TestGetInvokeStrategy(t *testing.T) {
 			expectError: false,
 		},
 		{
-			// case: change specializationtimeout
+			name: "change specializationtimeout",
 			testArgs: map[string]interface{}{
-				flagkey.FnExecutorType:          fv1.ExecutorTypeNewdeploy,
+				flagkey.FnExecutorType:          string(fv1.ExecutorTypeNewdeploy),
 				flagkey.FnSpecializationTimeout: 200,
 			},
 			existingInvokeStrategy: &fv1.InvokeStrategy{
@@ -316,19 +348,9 @@ func TestGetInvokeStrategy(t *testing.T) {
 			expectError: false,
 		},
 		{
-			// case: specializationtimeout should not work for poolmgr
+			name: "specializationtimeout should not be less than 120",
 			testArgs: map[string]interface{}{
-				flagkey.FnExecutorType:          fv1.ExecutorTypePoolmgr,
+				flagkey.FnExecutorType:          string(fv1.ExecutorTypeNewdeploy),
-				flagkey.FnSpecializationTimeout: 10,
-			},
-			existingInvokeStrategy: nil,
-			expectedResult:         nil,
-			expectError:            true,
-		},
-		{
-			// case: specializationtimeout should not be less than 120
-			testArgs: map[string]interface{}{
-				flagkey.FnExecutorType:          fv1.ExecutorTypeNewdeploy,
 				flagkey.FnSpecializationTimeout: 90,
 			},
 			existingInvokeStrategy: nil,
@@ -337,25 +359,27 @@ func TestGetInvokeStrategy(t *testing.T) {
 		},
 	}
 
-	for i, c := range cases {
+	for _, c := range cases {
-		fmt.Printf("=== Test Case %v ===\n", i)
+		t.Run(c.name, func(t *testing.T) {
+			flags := dummy.TestFlagSet()
 
-		flags := dummy.TestFlagSet()
+			for k, v := range c.testArgs {
-
+				flags.Set(k, v)
-		for k, v := range c.testArgs {
-			flags.Set(k, v)
-		}
-
-		strategy, err := getInvokeStrategy(flags, c.existingInvokeStrategy)
-		if c.expectError {
-			assert.NotNil(t, err)
-			if err != nil {
-				fmt.Println(err)
 			}
-		} else {
+
-			assert.Nil(t, err)
+			strategy, err := getInvokeStrategy(flags, c.existingInvokeStrategy)
-			assert.NoError(t, strategy.Validate(), fmt.Sprintf("Failed at test case %v", i))
+			if c.expectError {
-			assert.Equal(t, *c.expectedResult, *strategy)
+				assert.NotNil(t, err)
-		}
+				if err != nil {
+					fmt.Println(err)
+				}
+			} else {
+				assert.Nil(t, err)
+				if err == nil {
+					assert.NoError(t, strategy.Validate())
+					assert.Equal(t, *c.expectedResult, *strategy)
+				}
+			}
+		})
 	}
 }

pkg/fission-cli/cmd/function/test.go

@@ -31,6 +31,7 @@ import (
 	"github.com/fission/fission/pkg/controller/client"
 	"github.com/fission/fission/pkg/fission-cli/cliwrapper/cli"
 	"github.com/fission/fission/pkg/fission-cli/cmd/httptrigger"
+	"github.com/fission/fission/pkg/fission-cli/console"
 	flagkey "github.com/fission/fission/pkg/fission-cli/flag/key"
 	"github.com/fission/fission/pkg/fission-cli/util"
 )
@@ -65,6 +66,7 @@ func (opts *TestSubCommand) do(input cli.Input) error {
 		}
 		routerURL = "127.0.0.1:" + localRouterPort
 	} else {
+		console.Verbose(2, "Env FISSION_ROUTER: %v", routerURL)
 		routerURL = strings.TrimPrefix(routerURL, "http://")
 	}
 
@@ -77,6 +79,9 @@ func (opts *TestSubCommand) do(input cli.Input) error {
 	if err != nil {
 		return err
 	}
+
+	console.Verbose(2, "Function test url: %v", functionUrl.String())
+
 	queryParams := input.StringSlice(flagkey.FnTestQuery)
 	if len(queryParams) > 0 {
 		query := url.Values{}
@@ -97,16 +102,13 @@ func (opts *TestSubCommand) do(input cli.Input) error {
 		functionUrl.RawQuery = query.Encode()
 	}
 
-	ctx := context.Background()
+	ctx, closeCtx := context.WithTimeout(context.Background(), input.Duration(flagkey.FnTestTimeout))
-	if deadline := input.Duration(flagkey.FnTestTimeout); deadline > 0 {
+	defer closeCtx()
-		var closeCtx func()
-		ctx, closeCtx = context.WithTimeout(ctx, deadline)
-		defer closeCtx()
-	}
 
-	headers := input.StringSlice(flagkey.FnTestHeader)
+	resp, err := doHTTPRequest(ctx, functionUrl.String(),
-
+		input.StringSlice(flagkey.FnTestHeader),
-	resp, err := doHTTPRequest(ctx, input.String(flagkey.HtMethod), functionUrl.String(), input.String(flagkey.FnTestBody), headers)
+		input.String(flagkey.HtMethod),
+		input.String(flagkey.FnTestBody))
 	if err != nil {
 		return err
 	}
@@ -118,21 +120,25 @@ func (opts *TestSubCommand) do(input cli.Input) error {
 	}
 
 	if resp.StatusCode < 400 {
-		fmt.Print(string(body))
+		os.Stdout.Write(body)
 		return nil
 	}
 
-	fmt.Printf("Error calling function %s: %d; Please try again or fix the error: %s", m.Name, resp.StatusCode, string(body))
+	console.Errorf("Error calling function %s: %d; Please try again or fix the error: %s\n", m.Name, resp.StatusCode, string(body))
-	err = printPodLogs(input)
+	log, err := printPodLogs(opts.client, m)
 	if err != nil {
-		fmt.Printf("Error getting function logs from pod: %v. Try to get logs from log database", err)
+		console.Errorf("Error getting function logs from controller: %v. Try to get logs from log database.", err)
-		return Log(input)
+		err = Log(input)
+		if err != nil {
+			return errors.Wrapf(err, "error retrieving function log from log database")
+		}
+	} else {
+		console.Info(log)
 	}
-
+	return errors.New("error getting function response")
-	return nil
 }
 
-func doHTTPRequest(ctx context.Context, method, url, body string, headers []string) (*http.Response, error) {
+func doHTTPRequest(ctx context.Context, url string, headers []string, method, body string) (*http.Response, error) {
 	method, err := httptrigger.GetMethod(method)
 	if err != nil {
 		return nil, err
@@ -158,41 +164,21 @@ func doHTTPRequest(ctx context.Context, method, url, body string, headers []stri
 	return resp, nil
 }
 
-func printPodLogs(input cli.Input) error {
+func printPodLogs(client *client.Client, fnMeta *metav1.ObjectMeta) (string, error) {
-	fnName := input.String(flagkey.FnName)
+	reader, statusCode, err := client.FunctionPodLogs(fnMeta)
-
-	u, err := util.GetApplicationUrl("application=fission-api")
 	if err != nil {
-		return err
+		return "", errors.Wrap(err, "error executing get logs request")
 	}
+	defer reader.Close()
 
-	queryURL, err := url.Parse(u)
+	body, err := ioutil.ReadAll(reader)
 	if err != nil {
-		return errors.Wrap(err, "error parsing the base URL")
+		return "", errors.Wrap(err, "error reading the response body")
-	}
-	queryURL.Path = fmt.Sprintf("/proxy/logs/%s", fnName)
-
-	req, err := http.NewRequest(http.MethodPost, queryURL.String(), nil)
-	if err != nil {
-		return errors.Wrap(err, "error creating logs request")
 	}
 
-	httpClient := http.Client{}
+	if statusCode != http.StatusOK {
-	resp, err := httpClient.Do(req)
+		return string(body), errors.Errorf("error getting logs from controller, status code: '%v'", statusCode)
-	if err != nil {
-		return errors.Wrap(err, "execute get logs request")
 	}
 
-	defer resp.Body.Close()
+	return string(body), nil
-	if resp.StatusCode != http.StatusOK {
-		return errors.New("get logs from pod directly")
-	}
-
-	body, err := ioutil.ReadAll(resp.Body)
-	if err != nil {
-		return errors.Wrap(err, "read the response body")
-	}
-
-	fmt.Println(string(body))
-	return nil
 }

pkg/fission-cli/cmd/function/update.go

@@ -30,7 +30,6 @@ import (
 	"github.com/fission/fission/pkg/fission-cli/console"
 	flagkey "github.com/fission/fission/pkg/fission-cli/flag/key"
 	"github.com/fission/fission/pkg/fission-cli/util"
-	"github.com/fission/fission/pkg/types"
 )
 
 type UpdateSubCommand struct {
@@ -82,31 +81,12 @@ func (opts *UpdateSubCommand) complete(input cli.Input) error {
 		envNamespace = ""
 	}
 
-	var deployArchiveFiles []string
-	codeFlag := false
-	code := input.String(flagkey.PkgCode)
-	if len(code) == 0 {
-		deployArchiveFiles = input.StringSlice(flagkey.PkgDeployArchive)
-	} else {
-		deployArchiveFiles = append(deployArchiveFiles, input.String(flagkey.PkgCode))
-		codeFlag = true
-	}
-
-	srcArchiveFiles := input.StringSlice(flagkey.PkgSrcArchive)
 	pkgName := input.String(flagkey.FnPackageName)
 	entrypoint := input.String(flagkey.FnEntrypoint)
-	buildcmd := input.String(flagkey.PkgBuildCmd)
-	force := input.Bool(flagkey.PkgForce)
 
 	secretNames := input.StringSlice(flagkey.FnSecret)
 	cfgMapNames := input.StringSlice(flagkey.FnCfgMap)
 
-	specializationTimeout := input.Int(flagkey.FnSpecializationTimeout)
-
-	if len(srcArchiveFiles) > 0 && len(deployArchiveFiles) > 0 {
-		return errors.Errorf("need either of --%v or --%v and not both arguments", flagkey.PkgSrcArchive, flagkey.PkgDeployArchive)
-	}
-
 	var secrets []fv1.SecretReference
 	var configMaps []fv1.ConfigMapReference
 
@@ -187,18 +167,6 @@ func (opts *UpdateSubCommand) complete(input cli.Input) error {
 	}
 	function.Spec.InvokeStrategy = *strategy
 
-	if input.IsSet(flagkey.FnSpecializationTimeout) {
-		if strategy.ExecutionStrategy.ExecutorType != types.ExecutorTypeNewdeploy {
-			return errors.Errorf("--%v flag is only applicable for newdeploy type of executor", flagkey.FnSpecializationTimeout)
-		}
-
-		if specializationTimeout < fv1.DefaultSpecializationTimeOut {
-			return errors.Errorf("--%v must be greater than or equal to 120 seconds", flagkey.FnSpecializationTimeout)
-		} else {
-			function.Spec.InvokeStrategy.ExecutionStrategy.SpecializationTimeout = specializationTimeout
-		}
-	}
-
 	resReqs, err := util.GetResourceReqs(input, &function.Spec.Resources)
 	if err != nil {
 		return err
@@ -214,36 +182,38 @@ func (opts *UpdateSubCommand) complete(input cli.Input) error {
 		return errors.Wrap(err, fmt.Sprintf("read package '%v.%v'. Pkg should be present in the same ns as the function", pkgName, fnNamespace))
 	}
 
-	pkgMetadata := &pkg.Metadata
+	forceUpdate := input.Bool(flagkey.PkgForce)
 
-	if len(deployArchiveFiles) != 0 || len(srcArchiveFiles) != 0 || len(buildcmd) != 0 || len(envName) != 0 || len(envNamespace) != 0 {
+	fnList, err := _package.GetFunctionsByPackage(opts.client, pkg.Metadata.Name, pkg.Metadata.Namespace)
-		fnList, err := _package.GetFunctionsByPackage(opts.client, pkg.Metadata.Name, pkg.Metadata.Namespace)
+	if err != nil {
-		if err != nil {
+		return errors.Wrap(err, "error getting function list")
-			return errors.Wrap(err, "error getting function list")
+	}
-		}
 
-		if !force && len(fnList) > 1 {
+	if !forceUpdate && len(fnList) > 1 {
-			return errors.New("package is used by multiple functions, use --force to force update")
+		return errors.Errorf("Package is used by multiple functions, use --%v to force update", flagkey.PkgForce)
-		}
+	}
 
-		pkgMetadata, err = _package.UpdatePackage(opts.client, pkg, envName, envNamespace, srcArchiveFiles, deployArchiveFiles, buildcmd, false, codeFlag)
+	newPkgMeta, err := _package.UpdatePackage(input, opts.client, pkg)
-		if err != nil {
+	if err != nil {
-			return errors.Wrap(err, fmt.Sprintf("error updating package '%v'", pkgName))
+		return errors.Wrap(err, fmt.Sprintf("error updating package '%v'", pkgName))
-		}
+	}
 
-		fmt.Printf("package '%v' updated\n", pkgMetadata.GetName())
+	// the package resource version of function has been changed,
-
+	// we need to update function resource version to prevent conflict.
-		// update resource version of package reference of functions that shared the same package
+	// TODO: remove this block when deprecating pkg flags of function command.
+	if pkg.Metadata.ResourceVersion != newPkgMeta.ResourceVersion {
+		var fns []fv1.Function
+		// don't update the package resource version of the function we are currently
+		// updating to prevent update conflict.
 		for _, fn := range fnList {
-			// ignore the update for current function here, it will be updated later.
+			if fn.Metadata.UID != function.Metadata.UID {
-			if fn.Metadata.Name != fnName {
+				fns = append(fns, fn)
-				fn.Spec.Package.PackageRef.ResourceVersion = pkgMetadata.ResourceVersion
-				_, err := opts.client.FunctionUpdate(&fn)
-				if err != nil {
-					return errors.Wrap(err, "error updating function")
-				}
 			}
 		}
+		err = _package.UpdateFunctionPackageResourceVersion(opts.client, newPkgMeta, fns...)
+		if err != nil {
+			return errors.Wrap(err, "error updating function package reference resource version")
+		}
 	}
 
 	// TODO : One corner case where user just updates the pkg reference with fnUpdate, but internally this new pkg reference
@@ -251,9 +221,9 @@ func (opts *UpdateSubCommand) complete(input cli.Input) error {
 
 	// update function spec with new package metadata
 	function.Spec.Package.PackageRef = fv1.PackageRef{
-		Namespace:       pkgMetadata.Namespace,
+		Namespace:       newPkgMeta.Namespace,
-		Name:            pkgMetadata.Name,
+		Name:            newPkgMeta.Name,
-		ResourceVersion: pkgMetadata.ResourceVersion,
+		ResourceVersion: newPkgMeta.ResourceVersion,
 	}
 
 	if function.Spec.Environment.Name != pkg.Spec.Environment.Name {
@@ -273,6 +243,6 @@ func (opts *UpdateSubCommand) run(input cli.Input) error {
 		return errors.Wrap(err, "error updating function")
 	}
 
-	fmt.Printf("function '%v' updated\n", opts.function.Metadata.Name)
+	fmt.Printf("Function '%v' updated\n", opts.function.Metadata.Name)
 	return nil
 }

pkg/fission-cli/cmd/package/command.go

@@ -32,7 +32,8 @@ func Commands() *cobra.Command {
 	wrapper.SetFlags(createCmd, flag.FlagSet{
 		Required: []flag.Flag{flag.PkgEnvironment},
 		Optional: []flag.Flag{flag.PkgName, flag.PkgCode, flag.PkgSrcArchive, flag.PkgDeployArchive,
-			flag.PkgBuildCmd, flag.NamespacePackage, flag.NamespaceEnvironment, flag.SpecSave},
+			flag.PkgSrcChecksum, flag.PkgDeployChecksum, flag.PkgInsecure, flag.PkgBuildCmd,
+			flag.NamespacePackage, flag.NamespaceEnvironment, flag.SpecSave},
 	})
 
 	getSrcCmd := &cobra.Command{
@@ -62,8 +63,9 @@ func Commands() *cobra.Command {
 	}
 	wrapper.SetFlags(updateCmd, flag.FlagSet{
 		Required: []flag.Flag{flag.PkgName},
-		Optional: []flag.Flag{flag.PkgEnvironment, flag.PkgSrcArchive, flag.PkgDeployArchive,
+		Optional: []flag.Flag{flag.PkgEnvironment, flag.PkgCode, flag.PkgSrcArchive, flag.PkgDeployArchive,
-			flag.PkgBuildCmd, flag.PkgForce, flag.NamespacePackage, flag.NamespaceEnvironment},
+			flag.PkgSrcChecksum, flag.PkgDeployChecksum, flag.PkgInsecure, flag.PkgBuildCmd, flag.PkgForce,
+			flag.NamespacePackage, flag.NamespaceEnvironment},
 	})
 
 	deleteCmd := &cobra.Command{

pkg/fission-cli/cmd/package/create.go

@@ -124,6 +124,10 @@ func (opts *CreateSubCommand) run(input cli.Input) error {
 func CreatePackage(input cli.Input, client *client.Client, pkgName string, pkgNamespace string, envName string, envNamespace string,
 	srcArchiveFiles []string, deployArchiveFiles []string, buildcmd string, specDir string, specFile string, noZip bool) (*metav1.ObjectMeta, error) {
 
+	insecure := input.Bool(flagkey.PkgInsecure)
+	deployChecksum := input.String(flagkey.PkgDeployChecksum)
+	srcChecksum := input.String(flagkey.PkgSrcChecksum)
+
 	pkgSpec := fv1.PackageSpec{
 		Environment: fv1.EnvironmentReference{
 			Namespace: envNamespace,
@@ -136,9 +140,9 @@ func CreatePackage(input cli.Input, client *client.Client, pkgName string, pkgNa
 		if len(specFile) > 0 { // we should do this in all cases, i think
 			pkgStatus = fv1.BuildStatusNone
 		}
-		deployment, err := CreateArchive(client, deployArchiveFiles, noZip, specDir, specFile)
+		deployment, err := CreateArchive(client, deployArchiveFiles, noZip, insecure, deployChecksum, specDir, specFile)
 		if err != nil {
-			return nil, err
+			return nil, errors.Wrap(err, "error creating source archive")
 		}
 		pkgSpec.Deployment = *deployment
 		if len(pkgName) == 0 {
@@ -146,9 +150,9 @@ func CreatePackage(input cli.Input, client *client.Client, pkgName string, pkgNa
 		}
 	}
 	if len(srcArchiveFiles) > 0 {
-		source, err := CreateArchive(client, srcArchiveFiles, false, specDir, specFile)
+		source, err := CreateArchive(client, srcArchiveFiles, false, insecure, srcChecksum, specDir, specFile)
 		if err != nil {
-			return nil, err
+			return nil, errors.Wrap(err, "error creating deploy archive")
 		}
 		pkgSpec.Source = *source
 		pkgStatus = fv1.BuildStatusPending // set package build status to pending
@@ -164,6 +168,7 @@ func CreatePackage(input cli.Input, client *client.Client, pkgName string, pkgNa
 	if len(pkgName) == 0 {
 		pkgName = strings.ToLower(uuid.NewV4().String())
 	}
+
 	pkg := &fv1.Package{
 		Metadata: metav1.ObjectMeta{
 			Name:      pkgName,

pkg/fission-cli/cmd/package/package.go

@@ -19,6 +19,7 @@ package _package
 import (
 	"context"
 	"fmt"
+	"net/http"
 	"os"
 	"path/filepath"
 	"strings"
@@ -33,15 +34,18 @@ import (
 	pkgutil "github.com/fission/fission/pkg/fission-cli/cmd/package/util"
 	"github.com/fission/fission/pkg/fission-cli/cmd/spec"
 	spectypes "github.com/fission/fission/pkg/fission-cli/cmd/spec/types"
+	"github.com/fission/fission/pkg/fission-cli/console"
+	flagkey "github.com/fission/fission/pkg/fission-cli/flag/key"
 	"github.com/fission/fission/pkg/fission-cli/util"
 	"github.com/fission/fission/pkg/utils"
+	uuid "github.com/satori/go.uuid"
 )
 
 // CreateArchive returns a fv1.Archive made from an archive .  If specFile, then
 // create an archive upload spec in the specs directory; otherwise
 // upload the archive using client.  noZip avoids zipping the
 // includeFiles, but is ignored if there's more than one includeFile.
-func CreateArchive(client *client.Client, includeFiles []string, noZip bool, specDir string, specFile string) (*fv1.Archive, error) {
+func CreateArchive(client *client.Client, includeFiles []string, noZip bool, insecure bool, checksum string, specDir string, specFile string) (*fv1.Archive, error) {
 	// get root dir
 	var rootDir string
 	var err error
@@ -81,7 +85,7 @@ func CreateArchive(client *client.Client, includeFiles []string, noZip bool, spe
 		}
 
 		path := filepath.Join(rootDir, path)
-		files, err := utils.FindAllGlobs([]string{path})
+		files, err := utils.FindAllGlobs(path)
 		if err != nil {
 			errs = multierror.Append(errs, errors.Wrap(err, "error finding all globs"))
 			continue
@@ -96,51 +100,80 @@ func CreateArchive(client *client.Client, includeFiles []string, noZip bool, spe
 		return nil, errs.ErrorOrNil()
 	}
 
-	if len(specFile) > 0 {
+	if len(fileURL) > 0 {
-		if len(fileURL) > 0 {
+		if insecure {
 			return &fv1.Archive{
 				Type: fv1.ArchiveTypeUrl,
 				URL:  fileURL,
 			}, nil
-		} else {
-			// create an ArchiveUploadSpec and reference it from the archive
-			aus := &spectypes.ArchiveUploadSpec{
-				Name:         archiveName("", includeFiles),
-				IncludeGlobs: includeFiles,
-			}
-
-			// check if this AUS exists in the specs; if so, don't create a new one
-			fr, err := spec.ReadSpecs(specDir)
-			if err != nil {
-				return nil, errors.Wrap(err, "error reading specs")
-			}
-
-			obj := fr.SpecExists(aus, true, true)
-			if obj != nil {
-				oldAus := obj.(*spectypes.ArchiveUploadSpec)
-				fmt.Printf("Re-using previously created archive %v\n", oldAus.Name)
-				aus.Name = oldAus.Name
-			} else {
-				// save the uploadspec
-				err := spec.SpecSave(*aus, specFile)
-				if err != nil {
-					return nil, errors.Wrapf(err, "write spec file %v", specFile)
-				}
-			}
-			// create the archive object
-			archive := fv1.Archive{
-				Type: fv1.ArchiveTypeUrl,
-				URL:  fmt.Sprintf("%v%v", spec.ARCHIVE_URL_PREFIX, aus.Name),
-			}
-			return &archive, nil
 		}
+
+		var csum *fv1.Checksum
+
+		if len(checksum) > 0 {
+			csum = &fv1.Checksum{
+				Type: fv1.ChecksumTypeSHA256,
+				Sum:  checksum,
+			}
+		} else {
+			console.Info(fmt.Sprintf("Downloading file to generate SHA256 checksum. To skip this step, please use --%v / --%v / --%v",
+				flagkey.PkgSrcChecksum, flagkey.PkgDeployChecksum, flagkey.PkgInsecure))
+
+			tmpDir, err := utils.GetTempDir()
+			if err != nil {
+				return nil, err
+			}
+
+			file := filepath.Join(tmpDir, uuid.NewV4().String())
+			err = utils.DownloadUrl(context.Background(), http.DefaultClient, fileURL, file)
+			if err != nil {
+				return nil, errors.Wrap(err, "error downloading file from the given URL")
+			}
+
+			csum, err = utils.GetFileChecksum(file)
+			if err != nil {
+				return nil, errors.Wrap(err, "error generating file SHA256 checksum")
+			}
+		}
+
+		return &fv1.Archive{
+			Type:     fv1.ArchiveTypeUrl,
+			URL:      fileURL,
+			Checksum: *csum,
+		}, nil
 	}
 
-	if len(fileURL) > 0 {
+	if len(specFile) > 0 {
-		return &fv1.Archive{
+		// create an ArchiveUploadSpec and reference it from the archive
+		aus := &spectypes.ArchiveUploadSpec{
+			Name:         archiveName("", includeFiles),
+			IncludeGlobs: includeFiles,
+		}
+
+		// check if this AUS exists in the specs; if so, don't create a new one
+		fr, err := spec.ReadSpecs(specDir)
+		if err != nil {
+			return nil, errors.Wrap(err, "error reading specs")
+		}
+
+		obj := fr.SpecExists(aus, true, true)
+		if obj != nil {
+			oldAus := obj.(*spectypes.ArchiveUploadSpec)
+			fmt.Printf("Re-using previously created archive %v\n", oldAus.Name)
+			aus.Name = oldAus.Name
+		} else {
+			// save the uploadspec
+			err := spec.SpecSave(*aus, specFile)
+			if err != nil {
+				return nil, errors.Wrapf(err, "write spec file %v", specFile)
+			}
+		}
+		// create the archive object
+		archive := fv1.Archive{
 			Type: fv1.ArchiveTypeUrl,
-			URL:  fileURL,
+			URL:  fmt.Sprintf("%v%v", spec.ARCHIVE_URL_PREFIX, aus.Name),
-		}, nil
+		}
+		return &archive, nil
 	}
 
 	archivePath, err := makeArchiveFile("", includeFiles, noZip)
@@ -165,7 +198,7 @@ func makeArchiveFile(archiveNameHint string, archiveInput []string, noZip bool)
 	archiveName := archiveName(archiveNameHint, archiveInput)
 
 	// Get files from inputs as number of files decide next steps
-	files, err := utils.FindAllGlobs(archiveInput)
+	files, err := utils.FindAllGlobs(archiveInput...)
 	if err != nil {
 		return "", errors.Wrap(err, "error finding all globs")
 	}

pkg/fission-cli/cmd/package/update.go

@@ -20,6 +20,7 @@ import (
 	"fmt"
 	"time"
 
+	"github.com/hashicorp/go-multierror"
 	"github.com/pkg/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
@@ -31,15 +32,10 @@ import (
 )
 
 type UpdateSubCommand struct {
-	client             *client.Client
+	client       *client.Client
-	pkgName            string
+	pkgName      string
-	pkgNamespace       string
+	pkgNamespace string
-	force              bool
+	force        bool
-	envName            string
-	envNamespace       string
-	srcArchiveFiles    []string
-	deployArchiveFiles []string
-	buildcmd           string
 }
 
 func Update(input cli.Input) error {
@@ -65,11 +61,6 @@ func (opts *UpdateSubCommand) complete(input cli.Input) error {
 	opts.pkgName = input.String(flagkey.PkgName)
 	opts.pkgNamespace = input.String(flagkey.NamespacePackage)
 	opts.force = input.Bool(flagkey.PkgForce)
-	opts.envName = input.String(flagkey.PkgEnvironment)
-	opts.envNamespace = input.String(flagkey.NamespaceEnvironment)
-	opts.srcArchiveFiles = input.StringSlice(flagkey.PkgSrcArchive)
-	opts.deployArchiveFiles = input.StringSlice(flagkey.PkgDeployArchive)
-	opts.buildcmd = input.String(flagkey.PkgBuildCmd)
 	return nil
 }
 
@@ -82,88 +73,111 @@ func (opts *UpdateSubCommand) run(input cli.Input) error {
 		return errors.Wrap(err, "get package")
 	}
 
-	// if the new env specified is the same as the old one, no need to update package
+	forceUpdate := input.Bool(flagkey.PkgForce)
-	// same is true for all update parameters, but, for now, we dont check all of them - because, its ok to
-	// re-write the object with same old values, we just end up getting a new resource version for the object.
-	if len(opts.envName) > 0 && opts.envName == pkg.Spec.Environment.Name {
-		opts.envName = ""
-	}
-
-	if opts.envNamespace == pkg.Spec.Environment.Namespace {
-		opts.envNamespace = ""
-	}
 
 	fnList, err := GetFunctionsByPackage(opts.client, pkg.Metadata.Name, pkg.Metadata.Namespace)
 	if err != nil {
-		return errors.Wrap(err, "get function list")
+		return errors.Wrap(err, "error getting function list")
 	}
 
-	if !opts.force && len(fnList) > 1 {
+	if !forceUpdate && len(fnList) > 1 {
-		return errors.New("Package is used by multiple functions, use --force to force update")
+		return errors.Errorf("package is used by multiple functions, use --%v to force update", flagkey.PkgForce)
 	}
 
-	newPkgMeta, err := UpdatePackage(opts.client, pkg,
+	newPkgMeta, err := UpdatePackage(input, opts.client, pkg)
-		opts.envName, opts.envNamespace, opts.srcArchiveFiles,
-		opts.deployArchiveFiles, opts.buildcmd, false, false)
 	if err != nil {
-		return errors.Wrap(err, "update package")
+		return errors.Wrap(err, "error updating package")
 	}
 
-	// update resource version of package reference of functions that shared the same package
+	if pkg.Metadata.ResourceVersion != newPkgMeta.ResourceVersion {
-	for _, fn := range fnList {
+		err = UpdateFunctionPackageResourceVersion(opts.client, newPkgMeta, fnList...)
-		fn.Spec.Package.PackageRef.ResourceVersion = newPkgMeta.ResourceVersion
-		_, err := opts.client.FunctionUpdate(&fn)
 		if err != nil {
-			return errors.Wrap(err, "update function")
+			return errors.Wrap(err, "error updating function package reference resource version")
 		}
 	}
 
-	fmt.Printf("Package '%v' updated\n", newPkgMeta.GetName())
 	return nil
 }
 
-func UpdatePackage(client *client.Client, pkg *fv1.Package, envName, envNamespace string,
+func UpdatePackage(input cli.Input, client *client.Client, pkg *fv1.Package) (*metav1.ObjectMeta, error) {
-	srcArchiveFiles []string, deployArchiveFiles []string, buildcmd string, forceRebuild bool, noZip bool) (*metav1.ObjectMeta, error) {
+	envName := input.String(flagkey.PkgEnvironment)
+	envNamespace := input.String(flagkey.NamespaceEnvironment)
+	srcArchiveFiles := input.StringSlice(flagkey.PkgSrcArchive)
+	deployArchiveFiles := input.StringSlice(flagkey.PkgDeployArchive)
+	buildcmd := input.String(flagkey.PkgBuildCmd)
+	insecure := input.Bool(flagkey.PkgInsecure)
+	deployChecksum := input.String(flagkey.PkgDeployChecksum)
+	srcChecksum := input.String(flagkey.PkgSrcChecksum)
+	code := input.String(flagkey.PkgCode)
 
-	needToBuild := false
+	noZip := false
+	needToRebuild := false
+	needToUpdate := false
 
-	if len(envName) > 0 {
+	if input.IsSet(flagkey.PkgCode) {
+		deployArchiveFiles = append(deployArchiveFiles, code)
+		noZip = true
+		needToUpdate = true
+	}
+
+	if input.IsSet(flagkey.PkgEnvironment) {
 		pkg.Spec.Environment.Name = envName
-		needToBuild = true
+		needToRebuild = true
+		needToUpdate = true
 	}
 
-	if len(envNamespace) > 0 {
+	if input.IsSet(flagkey.NamespaceEnvironment) {
 		pkg.Spec.Environment.Namespace = envNamespace
-		needToBuild = true
+		needToRebuild = true
+		needToUpdate = true
 	}
 
-	if len(buildcmd) > 0 {
+	if input.IsSet(flagkey.PkgBuildCmd) {
 		pkg.Spec.BuildCommand = buildcmd
-		needToBuild = true
+		needToRebuild = true
+		needToUpdate = true
 	}
 
-	if len(srcArchiveFiles) > 0 {
+	if input.IsSet(flagkey.PkgSrcArchive) {
-		srcArchive, err := CreateArchive(client, srcArchiveFiles, false, "", "")
+		srcArchive, err := CreateArchive(client, srcArchiveFiles, noZip, insecure, srcChecksum, "", "")
 		if err != nil {
-			return nil, err
+			return nil, errors.Wrap(err, "error creating source archive")
 		}
 		pkg.Spec.Source = *srcArchive
-		needToBuild = true
+		needToRebuild = true
+		needToUpdate = true
+	} else if input.IsSet(flagkey.PkgSrcChecksum) {
+		pkg.Spec.Source.Checksum = fv1.Checksum{
+			Type: fv1.ChecksumTypeSHA256,
+			Sum:  srcChecksum,
+		}
+		needToUpdate = true
 	}
 
-	if len(deployArchiveFiles) > 0 {
+	if input.IsSet(flagkey.PkgDeployArchive) || input.IsSet(flagkey.PkgCode) {
-		deployArchive, err := CreateArchive(client, deployArchiveFiles, noZip, "", "")
+		deployArchive, err := CreateArchive(client, deployArchiveFiles, noZip, insecure, deployChecksum, "", "")
 		if err != nil {
-			return nil, err
+			return nil, errors.Wrap(err, "error creating deploy archive")
 		}
 		pkg.Spec.Deployment = *deployArchive
 		// Users may update the env, envNS and deploy archive at the same time,
 		// but without the source archive. In this case, we should set needToBuild to false
-		needToBuild = false
+		needToRebuild = false
+		needToUpdate = true
+	} else if input.IsSet(flagkey.PkgDeployChecksum) {
+		pkg.Spec.Deployment.Checksum = fv1.Checksum{
+			Type: fv1.ChecksumTypeSHA256,
+			Sum:  srcChecksum,
+		}
+		needToUpdate = true
+	}
+
+	if !needToUpdate {
+		return &pkg.Metadata, nil
 	}
 
 	// Set package as pending status when needToBuild is true
-	if needToBuild || forceRebuild {
+	if needToRebuild {
 		// change into pending state to trigger package build
 		pkg.Status = fv1.PackageStatus{
 			BuildStatus:         fv1.BuildStatusPending,
@@ -176,9 +190,26 @@ func UpdatePackage(client *client.Client, pkg *fv1.Package, envName, envNamespac
 		return nil, errors.Wrap(err, "update package")
 	}
 
+	fmt.Printf("Package '%v' updated\n", newPkgMeta.GetName())
+
 	return newPkgMeta, err
 }
 
+func UpdateFunctionPackageResourceVersion(client *client.Client, pkgMeta *metav1.ObjectMeta, fnList ...fv1.Function) error {
+	errs := &multierror.Error{}
+
+	// update resource version of package reference of functions that shared the same package
+	for _, fn := range fnList {
+		fn.Spec.Package.PackageRef.ResourceVersion = pkgMeta.ResourceVersion
+		_, err := client.FunctionUpdate(&fn)
+		if err != nil {
+			errs = multierror.Append(errs, errors.Wrapf(err, "error updating package resource version of function '%v'", fn.Metadata.Name))
+		}
+	}
+
+	return errs.ErrorOrNil()
+}
+
 func updatePackageStatus(client *client.Client, pkg *fv1.Package, status fv1.BuildStatus) (*metav1.ObjectMeta, error) {
 	switch status {
 	case fv1.BuildStatusNone, fv1.BuildStatusPending, fv1.BuildStatusRunning, fv1.BuildStatusSucceeded, fv1.CanaryConfigStatusAborted:

pkg/fission-cli/cmd/spec/apply.go

@@ -427,13 +427,13 @@ func localArchiveFromSpec(specDir string, aus *spectypes.ArchiveUploadSpec) (*fv
 		files = append(files, aus.IncludeGlobs[0])
 	} else {
 		for _, relativeGlob := range aus.IncludeGlobs {
-			absGlob := rootDir + "/" + relativeGlob
+			absGlob := filepath.Join(rootDir, relativeGlob)
-			f, err := filepath.Glob(absGlob)
+			console.Verbose(2, "try to find globs in path '%v'", absGlob)
+			fs, err := utils.FindAllGlobs(absGlob)
 			if err != nil {
 				return nil, errors.Wrapf(err, "Invalid glob in archive %v: %v", aus.Name, relativeGlob)
 			}
-			files = append(files, f...)
+			files = append(files, fs...)
-			// xxx handle excludeGlobs here
 		}
 	}
 

pkg/fission-cli/cmd/spec/init.go

@@ -105,13 +105,20 @@ func (opts *InitSubCommand) complete(input cli.Input) error {
 func (opts *InitSubCommand) run(input cli.Input) error {
 	specDir := util.GetSpecDir(input)
 
+	readme := filepath.Join(specDir, "README")
+	config := filepath.Join(specDir, "fission-deployment-config.yaml")
+
+	if _, err := os.Stat(config); err == nil {
+		return errors.Errorf("Spec DeploymentConfig already exists in directory '%v'", specDir)
+	}
+
 	// Add a bit of documentation to the spec dir here
-	err := ioutil.WriteFile(filepath.Join(specDir, "README"), []byte(SPEC_README), 0644)
+	err := ioutil.WriteFile(readme, []byte(SPEC_README), 0644)
 	if err != nil {
 		return err
 	}
 
-	err = writeDeploymentConfig(specDir, opts.deployConfig)
+	err = writeDeploymentConfig(config, opts.deployConfig)
 	if err != nil {
 		return errors.Wrap(err, "error writing deployment config")
 	}
@@ -124,7 +131,7 @@ func (opts *InitSubCommand) run(input cli.Input) error {
 
 // writeDeploymentConfig serializes the DeploymentConfig to YAML and writes it to a new
 // fission-config.yaml in specDir.
-func writeDeploymentConfig(specDir string, dc *spectypes.DeploymentConfig) error {
+func writeDeploymentConfig(file string, dc *spectypes.DeploymentConfig) error {
 	y, err := yaml.Marshal(dc)
 	if err != nil {
 		return err
@@ -134,7 +141,7 @@ func writeDeploymentConfig(specDir string, dc *spectypes.DeploymentConfig) error
 		"# See the README in this directory for background and usage information.\n" +
 		"# Do not edit the UID below: that will break 'fission spec apply'\n")
 
-	err = ioutil.WriteFile(filepath.Join(specDir, "fission-deployment-config.yaml"), append(msg, y...), 0644)
+	err = ioutil.WriteFile(file, append(msg, y...), 0644)
 	if err != nil {
 		return err
 	}

pkg/fission-cli/cmd/spec/spec.go

@@ -297,35 +297,26 @@ func (fr *FissionResources) Validate(input cli.Input) error {
 	for _, p := range fr.Packages {
 		packages[MapKey(&p.Metadata)] = false
 
-		if strings.HasPrefix(p.Spec.Deployment.URL, ARCHIVE_URL_PREFIX) {
+		as := map[string]string{
-			// check archive refs from package
+			"source":     p.Spec.Source.URL,
-			aname := strings.TrimPrefix(p.Spec.Source.URL, ARCHIVE_URL_PREFIX)
+			"deployment": p.Spec.Deployment.URL,
-			if len(aname) > 0 {
-				if _, ok := archives[aname]; !ok {
-					result = multierror.Append(result, fmt.Errorf(
-						"%v: package '%v' references unknown source archive %v%v",
-						fr.SourceMap.Locations["Package"][p.Metadata.Namespace][p.Metadata.Name],
-						p.Metadata.Name,
-						ARCHIVE_URL_PREFIX,
-						aname))
-				} else {
-					archives[aname] = true
-				}
-			}
 		}
 
-		if strings.HasPrefix(p.Spec.Deployment.URL, ARCHIVE_URL_PREFIX) {
+		for archiveType, u := range as {
-			aname := strings.TrimPrefix(p.Spec.Deployment.URL, ARCHIVE_URL_PREFIX)
+			if strings.HasPrefix(u, ARCHIVE_URL_PREFIX) {
-			if len(aname) > 0 {
+				aname := strings.TrimPrefix(u, ARCHIVE_URL_PREFIX)
-				if _, ok := archives[aname]; !ok {
+				if len(aname) > 0 {
-					result = multierror.Append(result, fmt.Errorf(
+					if _, ok := archives[aname]; !ok {
-						"%v: package '%v' references unknown deployment archive %v%v",
+						result = multierror.Append(result, fmt.Errorf(
-						fr.SourceMap.Locations["Package"][p.Metadata.Namespace][p.Metadata.Name],
+							"%v: package '%v' references unknown %v archive '%v%v'",
-						p.Metadata.Name,
+							fr.SourceMap.Locations["Package"][p.Metadata.Namespace][p.Metadata.Name],
-						ARCHIVE_URL_PREFIX,
+							p.Metadata.Name,
-						aname))
+							archiveType,
-				} else {
+							ARCHIVE_URL_PREFIX,
-					archives[aname] = true
+							aname))
+					} else {
+						archives[aname] = true
+					}
 				}
 			}
 		}

pkg/fission-cli/console/log.go

@@ -33,6 +33,11 @@ func Error(msg interface{}) {
 	os.Stderr.WriteString(fmt.Sprintf("%v: %v\n", color.RedString("Error"), trimNewline(msg)))
 }
 
+func Errorf(format string, args ...interface{}) {
+	msg := fmt.Sprintf(format, args...)
+	os.Stderr.WriteString(fmt.Sprintf("%v: %v\n", color.RedString("Error"), trimNewline(msg)))
+}
+
 func Warn(msg interface{}) {
 	os.Stdout.WriteString(fmt.Sprintf("%v: %v\n", color.YellowString("Warning"), trimNewline(msg)))
 }

pkg/fission-cli/flag/flag.go

@@ -24,13 +24,13 @@ import (
 
 	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
 	flagkey "github.com/fission/fission/pkg/fission-cli/flag/key"
-	"github.com/fission/fission/pkg/types"
 )
 
 type (
 	FlagType = int
 
 	FlagSet struct {
+		Global   []Flag
 		Required []Flag
 		Optional []Flag
 	}
@@ -93,7 +93,7 @@ var (
 	FnBuildCmd              = Flag{Type: String, Name: flagkey.FnBuildCmd, Usage: "Package build command for builder to run with"}
 	FnSecret                = Flag{Type: StringSlice, Name: flagkey.FnSecret, Usage: "Function access to secret, should be present in the same namespace as the function. You can provide multiple secrets using multiple --secrets flags. In the case of fn update the the secrets will be replaced by the provided list of secrets."}
 	FnCfgMap                = Flag{Type: StringSlice, Name: flagkey.FnCfgMap, Usage: "Function access to configmap, should be present in the same namespace as the function. You can provide multiple configmaps using multiple --configmap flags. In case of fn update the configmaps will be replaced by the provided list of configmaps."}
-	FnExecutorType          = Flag{Type: String, Name: flagkey.FnExecutorType, Usage: "Executor type for execution; one of 'poolmgr', 'newdeploy'", DefaultValue: types.ExecutorTypePoolmgr}
+	FnExecutorType          = Flag{Type: String, Name: flagkey.FnExecutorType, Usage: "Executor type for execution; one of 'poolmgr', 'newdeploy'", DefaultValue: string(fv1.ExecutorTypePoolmgr)}
 	FnExecutionTimeout      = Flag{Type: Int, Name: flagkey.FnExecutionTimeout, Aliases: []string{"ft"}, Usage: "Maximum time for a request to wait for the response from the function", DefaultValue: 60}
 	FnLogPod                = Flag{Type: String, Name: flagkey.FnLogPod, Usage: "Function pod name (use the latest pod name if unspecified)"}
 	FnLogFollow             = Flag{Type: Bool, Name: flagkey.FnLogFollow, Short: "f", Usage: "Specify if the logs should be streamed"}
@@ -141,6 +141,7 @@ var (
 	EnvExternalNetwork        = Flag{Type: Bool, Name: flagkey.EnvExternalNetwork, Usage: "Allow pod to access external network (only works when istio feature is enabled)"}
 	EnvTerminationGracePeriod = Flag{Type: Int, Name: flagkey.EnvGracePeriod, Aliases: []string{"period"}, Usage: "Grace time (in seconds) for pod to perform connection draining before termination", DefaultValue: 360}
 	EnvVersion                = Flag{Type: Int, Name: flagkey.EnvVersion, Usage: "Environment API version (1 means v1 interface)", DefaultValue: 1}
+	EnvImagePullSecret        = Flag{Type: String, Name: flagkey.EnvImagePullSecret, Usage: "Secret for Kubernetes to pull an image from a private registry"}
 
 	KwName      = Flag{Type: String, Name: flagkey.KwName, Usage: "Watch name"}
 	KwFnName    = Flag{Type: String, Name: flagkey.KwFnName, Usage: "Function name"}
@@ -148,16 +149,19 @@ var (
 	KwObjType   = Flag{Type: String, Name: flagkey.KwObjType, Usage: "Type of resource to watch (Pod, Service, etc.)", DefaultValue: "pod"}
 	KwLabels    = Flag{Type: String, Name: flagkey.KwLabels, Usage: "Label selector of the form a=b,c=d"}
 
-	PkgName          = Flag{Type: String, Name: flagkey.PkgName, Usage: "Package name"}
+	PkgName           = Flag{Type: String, Name: flagkey.PkgName, Usage: "Package name"}
-	PkgForce         = Flag{Type: Bool, Name: flagkey.PkgForce, Short: "f", Usage: "Force update a package even if it is used by one or more functions"}
+	PkgForce          = Flag{Type: Bool, Name: flagkey.PkgForce, Short: "f", Usage: "Force update a package even if it is used by one or more functions"}
-	PkgEnvironment   = Flag{Type: String, Name: flagkey.PkgEnvironment, Usage: "Environment name"}
+	PkgEnvironment    = Flag{Type: String, Name: flagkey.PkgEnvironment, Usage: "Environment name"}
-	PkgBuildCmd      = Flag{Type: String, Name: flagkey.PkgBuildCmd, Usage: "Build command for builder to run with"}
+	PkgBuildCmd       = Flag{Type: String, Name: flagkey.PkgBuildCmd, Usage: "Build command for builder to run with"}
-	PkgOutput        = Flag{Type: String, Name: flagkey.PkgOutput, Short: "o", Usage: "Output filename to save archive content"}
+	PkgOutput         = Flag{Type: String, Name: flagkey.PkgOutput, Short: "o", Usage: "Output filename to save archive content"}
-	PkgStatus        = Flag{Type: String, Name: flagkey.PkgStatus, Usage: `Filter packages by status`}
+	PkgStatus         = Flag{Type: String, Name: flagkey.PkgStatus, Usage: `Filter packages by status`}
-	PkgOrphan        = Flag{Type: Bool, Name: flagkey.PkgOrphan, Usage: "Orphan packages that are not referenced by any function"}
+	PkgOrphan         = Flag{Type: Bool, Name: flagkey.PkgOrphan, Usage: "Orphan packages that are not referenced by any function"}
-	PkgCode          = Flag{Type: String, Name: flagkey.PkgCode, Usage: "URL or local path for single file source code"}
+	PkgCode           = Flag{Type: String, Name: flagkey.PkgCode, Usage: "URL or local path for single file source code"}
-	PkgDeployArchive = Flag{Type: StringSlice, Name: flagkey.PkgDeployArchive, Aliases: []string{"deploy"}, Usage: "URL or local paths for binary archive"}
+	PkgDeployArchive  = Flag{Type: StringSlice, Name: flagkey.PkgDeployArchive, Aliases: []string{"deploy"}, Usage: "URL or local paths for binary archive"}
-	PkgSrcArchive    = Flag{Type: StringSlice, Name: flagkey.PkgSrcArchive, Aliases: []string{"source", "src"}, Usage: "URL or local paths for source archive"}
+	PkgDeployChecksum = Flag{Type: String, Name: flagkey.PkgDeployChecksum, Usage: "SHA256 checksum of deploy archive when providing URL"}
+	PkgSrcArchive     = Flag{Type: StringSlice, Name: flagkey.PkgSrcArchive, Aliases: []string{"source", "src"}, Usage: "URL or local paths for source archive"}
+	PkgSrcChecksum    = Flag{Type: String, Name: flagkey.PkgSrcChecksum, Usage: "SHA256 checksum of source archive when providing URL"}
+	PkgInsecure       = Flag{Type: Bool, Name: flagkey.PkgInsecure, Usage: "Skip generating SHA256 checksum for file integrity validation"}
 
 	SpecSave     = Flag{Type: Bool, Name: flagkey.SpecSave, Usage: "Save to the spec directory instead of creating on cluster"}
 	SpecDir      = Flag{Type: String, Name: flagkey.SpecDir, Usage: "Directory to store specs, defaults to ./specs"}

pkg/fission-cli/flag/key/key.go

@@ -96,6 +96,7 @@ const (
 	EnvExternalNetwork = "externalnetwork"
 	EnvGracePeriod     = "graceperiod"
 	EnvVersion         = "version"
+	EnvImagePullSecret = "imagepullsecret"
 
 	KwName      = resourceName
 	KwFnName    = "function"
@@ -103,16 +104,19 @@ const (
 	KwObjType   = "type"
 	KwLabels    = "labels"
 
-	PkgName          = resourceName
+	PkgName           = resourceName
-	PkgForce         = force
+	PkgForce          = force
-	PkgEnvironment   = "env"
+	PkgEnvironment    = "env"
-	PkgCode          = "code"
+	PkgCode           = "code"
-	PkgSrcArchive    = "sourcearchive"
+	PkgSrcArchive     = "sourcearchive"
-	PkgDeployArchive = "deployarchive"
+	PkgDeployArchive  = "deployarchive"
-	PkgBuildCmd      = "buildcmd"
+	PkgSrcChecksum    = "srcchecksum"
-	PkgOutput        = Output
+	PkgDeployChecksum = "deploychecksum"
-	PkgStatus        = "status"
+	PkgInsecure       = "insecure"
-	PkgOrphan        = "orphan"
+	PkgBuildCmd       = "buildcmd"
+	PkgOutput         = Output
+	PkgStatus         = "status"
+	PkgOrphan         = "orphan"
 
 	SpecSave     = "spec"
 	SpecDir      = "specdir"

pkg/router/functionHandler.go

@@ -32,7 +32,6 @@ import (
 	"github.com/pkg/errors"
 	"go.opencensus.io/plugin/ochttp"
 	"go.uber.org/zap"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	k8stypes "k8s.io/apimachinery/pkg/types"
 
 	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
@@ -54,9 +53,9 @@ type (
 		logger                   *zap.Logger
 		fmap                     *functionServiceMap
 		executor                 *executorClient.Client
-		function                 *metav1.ObjectMeta
+		function                 *fv1.Function
 		httpTrigger              *fv1.HTTPTrigger
-		functionMetadataMap      map[string]*metav1.ObjectMeta
+		functionMap              map[string]*fv1.Function
 		fnWeightDistributionList []FunctionWeightDistribution
 		tsRoundTripperParams     *tsRoundTripperParams
 		isDebugEnv               bool
@@ -86,9 +85,13 @@ type (
 
 	// A layer on top of http.DefaultTransport, with retries.
 	RetryingRoundTripper struct {
-		logger      *zap.Logger
+		logger           *zap.Logger
-		funcHandler *functionHandler
+		funcHandler      *functionHandler
-		timeout     int
+		funcTimeout      time.Duration
+		closeContextFunc *context.CancelFunc
+		serviceUrl       *url.URL
+		urlFromCache     bool
+		totalRetry       int
 	}
 
 	// To keep the request body open during retries, we create an interface with Close operation being a no-op.
@@ -145,26 +148,10 @@ func (w *fakeCloseReadCloser) RealClose() error {
 // inside ServeHttp function of the reverseProxy.
 // Earlier, GetServiceForFunction was called inside handler function and fission explicitly set http status code to 500
 // if it returned an error.
-func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
+func (roundTripper *RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
 	// Set forwarded host header if not exists
 	roundTripper.addForwardedHostHeader(req)
 
-	fnMeta := roundTripper.funcHandler.function
-
-	// Metrics stuff
-	startTime := time.Now()
-	funcMetricLabels := &functionLabels{
-		namespace: fnMeta.Namespace,
-		name:      fnMeta.Name,
-	}
-	httpMetricLabels := &httpLabels{
-		method: req.Method,
-	}
-	if roundTripper.funcHandler.httpTrigger != nil {
-		httpMetricLabels.host = roundTripper.funcHandler.httpTrigger.Spec.Host
-		httpMetricLabels.path = roundTripper.funcHandler.httpTrigger.Spec.RelativeURL
-	}
-
 	// set the timeout for transport context
 	transport := roundTripper.getDefaultTransport()
 	ocRoundTripper := &ochttp.Transport{Base: transport}
@@ -183,6 +170,8 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
 		}
 	}()
 
+	roundTripper.logger.Debug("request headers", zap.Any("headers", req.Header))
+
 	// The reason for request failure may vary from case to case.
 	// After some investigation, found most of the failure are due to
 	// network timeout or target function is under heavy workload. In
@@ -194,19 +183,15 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
 	// than the predefined threshold, reset retryCounter and remove service
 	// cache, then retry to get new svc record from executor again.
 	var retryCounter int
-
-	var serviceUrl *url.URL
-	var serviceUrlFromCache bool
 	var err error
-
+	var fnMeta = &roundTripper.funcHandler.function.Metadata
-	var resp *http.Response
 
 	for i := 0; i < roundTripper.funcHandler.tsRoundTripperParams.maxRetries; i++ {
 		// set service url of target service of request only when
 		// trying to get new service url from cache/executor.
 		if retryCounter == 0 {
 			// get function service url from cache or executor
-			serviceUrl, serviceUrlFromCache, err = roundTripper.funcHandler.getServiceEntry()
+			roundTripper.serviceUrl, roundTripper.urlFromCache, err = roundTripper.funcHandler.getServiceEntry()
 			if err != nil {
 				// We might want a specific error code or header for fission failures as opposed to
 				// user function bugs.
@@ -228,21 +213,16 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
 
 			// service url maybe nil if router cannot find one in cache,
 			// so here we retry to get service url again
-			if serviceUrl == nil {
+			if roundTripper.serviceUrl == nil {
 				time.Sleep(executingTimeout)
 				executingTimeout = executingTimeout * time.Duration(roundTripper.funcHandler.tsRoundTripperParams.timeoutExponent)
 				continue
 			}
 
-			// tapService before invoking roundTrip for the serviceUrl
-			if serviceUrlFromCache {
-				go roundTripper.funcHandler.tapService(serviceUrl)
-			}
-
 			// modify the request to reflect the service url
 			// this service url may have come from the cache lookup or from executor response
-			req.URL.Scheme = serviceUrl.Scheme
+			req.URL.Scheme = roundTripper.serviceUrl.Scheme
-			req.URL.Host = serviceUrl.Host
+			req.URL.Host = roundTripper.serviceUrl.Host
 
 			// To keep the function run container simple, it
 			// doesn't do any routing.  In the future if we have
@@ -254,7 +234,7 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
 			// Overwrite request host with internal host,
 			// or request will be blocked in some situations
 			// (e.g. istio-proxy)
-			req.Host = serviceUrl.Host
+			req.Host = roundTripper.serviceUrl.Host
 		}
 
 		// over-riding default settings.
@@ -263,37 +243,21 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
 			KeepAlive: roundTripper.funcHandler.tsRoundTripperParams.keepAliveTime,
 		}).DialContext
 
-		overhead := time.Since(startTime)
+		// Do NOT assign returned request to "req"
-
+		// because the request used in the last round
-		roundTripper.logger.Debug("request headers", zap.Any("headers", req.Header))
+		// will be canceled when calling setContext.
-
+		newReq := roundTripper.setContext(req)
-		// Creating context for client
-		if roundTripper.timeout <= 0 {
-			roundTripper.timeout = fv1.DEFAULT_FUNCTION_TIMEOUT
-		}
-
-		roundTripper.logger.Debug("Creating context for request for ", zap.Any("time", roundTripper.timeout))
-		// pass request context as parent context for the case
-		// that user aborts connection before timeout. Otherwise,
-		// the request won't be canceled until the deadline exceeded
-		// which may be a potential security issue.
-		ctx, closeCtx := context.WithTimeout(req.Context(), time.Duration(roundTripper.timeout)*time.Second)
 
 		// forward the request to the function service
-		resp, err = ocRoundTripper.RoundTrip(req.WithContext(ctx))
+		resp, err := ocRoundTripper.RoundTrip(newReq)
-		closeCtx()
-
 		if err == nil {
-			// Track metrics
-			httpMetricLabels.code = resp.StatusCode
-			funcMetricLabels.cached = serviceUrlFromCache
-
-			functionCallCompleted(funcMetricLabels, httpMetricLabels,
-				overhead, time.Since(startTime), resp.ContentLength)
-
 			// return response back to user
 			return resp, nil
-		} else if i >= roundTripper.funcHandler.tsRoundTripperParams.maxRetries-1 {
+		}
+
+		roundTripper.totalRetry += 1
+
+		if i >= roundTripper.funcHandler.tsRoundTripperParams.maxRetries-1 {
 			// return here if we are in the last round
 			roundTripper.logger.Error("error getting response from function",
 				zap.String("function_name", fnMeta.Name),
@@ -316,11 +280,16 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
 			return resp, err
 		}
 
+		// close response body before entering next loop
+		if resp != nil {
+			resp.Body.Close()
+		}
+
 		// Check whether an error is an timeout error ("dial tcp i/o timeout").
 		// If it's not a timeout error or retryCounter exceeded pre-defined threshold,
 		// we assume the entry in router cache is stale, invalidate it.
 		if !isNetTimeoutErr || retryCounter >= roundTripper.funcHandler.tsRoundTripperParams.svcAddrRetryCount {
-			if serviceUrlFromCache {
+			if roundTripper.urlFromCache {
 				// if transport.RoundTrip returns a network dial error and serviceUrl was from cache,
 				// it means, the entry in router cache is stale, so invalidate it.
 				roundTripper.logger.Debug("request errored out - removing function from router's cache and requesting a new service for function",
@@ -342,11 +311,6 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
 		roundTripper.logger.Debug("Backing off before retrying", zap.Any("backoff_time", executingTimeout), zap.Error(err))
 		time.Sleep(executingTimeout)
 		executingTimeout = executingTimeout * time.Duration(roundTripper.funcHandler.tsRoundTripperParams.timeoutExponent)
-
-		// close response body before entering next loop
-		if resp != nil {
-			resp.Body.Close()
-		}
 	}
 
 	e := errors.New("Unable to get service url for connection")
@@ -373,25 +337,47 @@ func (roundTripper RetryingRoundTripper) getDefaultTransport() *http.Transport {
 	}
 }
 
-func (fh *functionHandler) tapService(serviceUrl *url.URL) {
+// setContext returns a shallow copy of request with a new timeout context.
+func (roundTripper *RetryingRoundTripper) setContext(req *http.Request) *http.Request {
+	if roundTripper.closeContextFunc != nil {
+		(*roundTripper.closeContextFunc)()
+	}
+	// pass request context as parent context for the case
+	// that user aborts connection before timeout. Otherwise,
+	// the request won't be canceled until the deadline exceeded
+	// which may be a potential security issue.
+	ctx, closeCtx := context.WithTimeout(req.Context(), roundTripper.funcTimeout)
+	roundTripper.closeContextFunc = &closeCtx
+
+	return req.WithContext(ctx)
+}
+
+// closeContext closes the context to release resources.
+func (roundTripper *RetryingRoundTripper) closeContext() {
+	if roundTripper.closeContextFunc != nil {
+		(*roundTripper.closeContextFunc)()
+	}
+}
+
+func (fh *functionHandler) tapService(fn *fv1.Function, serviceUrl *url.URL) {
 	if fh.executor == nil {
 		return
 	}
-	fh.executor.TapService(serviceUrl)
+	fh.executor.TapService(fn.Metadata, fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType, serviceUrl)
 }
 
 func (fh functionHandler) handler(responseWriter http.ResponseWriter, request *http.Request) {
 	if fh.httpTrigger != nil && fh.httpTrigger.Spec.FunctionReference.Type == types.FunctionReferenceTypeFunctionWeights {
 		// canary deployment. need to determine the function to send request to now
-		fnMetadata := getCanaryBackend(fh.functionMetadataMap, fh.fnWeightDistributionList)
+		fn := getCanaryBackend(fh.functionMap, fh.fnWeightDistributionList)
-		if fnMetadata == nil {
+		if fn == nil {
 			fh.logger.Error("could not get canary backend",
-				zap.Any("metadataMap", fh.functionMetadataMap),
+				zap.Any("fnMap", fh.functionMap),
 				zap.Any("distributionList", fh.fnWeightDistributionList))
 			// TODO : write error to responseWrite and return response
 			return
 		}
-		fh.function = fnMetadata
+		fh.function = fn
 		fh.logger.Debug("chosen function backend's metadata", zap.Any("metadata", fh.function))
 	}
 
@@ -399,7 +385,7 @@ func (fh functionHandler) handler(responseWriter http.ResponseWriter, request *h
 	setPathInfoToHeader(request)
 
 	// system params
-	setFunctionMetadataToHeader(fh.function, request)
+	setFunctionMetadataToHeader(&fh.function.Metadata, request)
 
 	director := func(req *http.Request) {
 		if _, ok := req.Header["User-Agent"]; !ok {
@@ -408,21 +394,42 @@ func (fh functionHandler) handler(responseWriter http.ResponseWriter, request *h
 		}
 	}
 
-	var timeout int = fv1.DEFAULT_FUNCTION_TIMEOUT
+	fnTimeout := fh.functionTimeoutMap[fh.function.Metadata.GetUID()]
-	if fh.functionTimeoutMap != nil {
+	if fnTimeout == 0 {
-		timeout = fh.functionTimeoutMap[fh.function.GetUID()]
+		fnTimeout = fv1.DEFAULT_FUNCTION_TIMEOUT
 	}
 
-	proxy := &httputil.ReverseProxy{
+	rrt := &RetryingRoundTripper{
-		Director: director,
+		logger:      fh.logger.Named("roundtripper"),
-		Transport: &RetryingRoundTripper{
+		funcHandler: &fh,
-			logger:      fh.logger.Named("roundtripper"),
+		funcTimeout: time.Duration(fnTimeout) * time.Second,
-			funcHandler: &fh,
-			timeout:     timeout,
-		},
-		ErrorHandler: getProxyErrorHandler(fh.logger, fh.function),
 	}
 
+	start := time.Now()
+
+	proxy := &httputil.ReverseProxy{
+		Director:     director,
+		Transport:    rrt,
+		ErrorHandler: fh.getProxyErrorHandler(start, rrt),
+		ModifyResponse: func(resp *http.Response) error {
+			go fh.collectFunctionMetric(start, rrt, request, resp)
+			return nil
+		},
+	}
+
+	defer func() {
+		// If the context is closed when RoundTrip returns, client may receive
+		// truncated response body due to "context canceled" error. To avoid
+		// this, we need to close request context after proxy.ServeHTTP finished.
+		//
+		// NOTE: rrt.closeContext() must be put in the defer function; otherwise,
+		// reverseProxy may panic when failed to write response and the context
+		// will not be closed.
+		//
+		// ref: https://github.com/golang/go/issues/28239
+		rrt.closeContext()
+	}()
+
 	proxy.ServeHTTP(responseWriter, request)
 }
 
@@ -453,38 +460,10 @@ func findCeil(randomNumber int, wtDistrList []FunctionWeightDistribution) string
 }
 
 // picks a function to route to based on a random number generated
-func getCanaryBackend(fnMetadatamap map[string]*metav1.ObjectMeta, fnWtDistributionList []FunctionWeightDistribution) *metav1.ObjectMeta {
+func getCanaryBackend(fnMap map[string]*fv1.Function, fnWtDistributionList []FunctionWeightDistribution) *fv1.Function {
 	randomNumber := rand.Intn(fnWtDistributionList[len(fnWtDistributionList)-1].sumPrefix + 1)
-
 	fnName := findCeil(randomNumber, fnWtDistributionList)
-
+	return fnMap[fnName]
-	return fnMetadatamap[fnName]
-}
-
-// getProxyErrorHandler returns a reverse proxy error handler
-func getProxyErrorHandler(logger *zap.Logger, fnMeta *metav1.ObjectMeta) func(rw http.ResponseWriter, req *http.Request, err error) {
-	return func(rw http.ResponseWriter, req *http.Request, err error) {
-		status := http.StatusBadGateway
-		switch err {
-		case context.Canceled:
-			// 499 CLIENT CLOSED REQUEST
-			// A non-standard status code introduced by nginx for the case
-			// when a client closes the connection while nginx is processing the request.
-			// Reference: https://httpstatuses.com/499
-			status = 499
-			logger.Debug("client closes the connection",
-				zap.Any("function", fnMeta), zap.Any("request_header", req.Header))
-		case context.DeadlineExceeded:
-			status = http.StatusGatewayTimeout
-			logger.Error("function not responses before the timeout",
-				zap.Any("function", fnMeta), zap.Any("request_header", req.Header))
-		default:
-			logger.Error("error sending request to function",
-				zap.Error(err), zap.Any("function", fnMeta), zap.Any("request_header", req.Header))
-		}
-		// TODO: return error message that contains traceable UUID back to user. Issue #693
-		rw.WriteHeader(status)
-	}
 }
 
 // addForwardedHostHeader add "forwarded host" to request header
@@ -544,28 +523,30 @@ func (fh *functionHandler) getServiceEntry() (serviceUrl *url.URL, serviceUrlFro
 	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
 	defer cancel()
 
+	fnMeta := &fh.function.Metadata
+
 	// Use throttle to limit the total amount of requests sent
 	// to the executor to prevent it from overloaded.
 	recordObj, err := fh.svcAddrUpdateThrottler.RunOnce(
-		crd.CacheKey(fh.function),
+		crd.CacheKey(fnMeta),
 		func(firstToTheLock bool) (interface{}, error) {
 			var u *url.URL
 			// Get service entry from executor and update cache if its the first goroutine
 			if firstToTheLock { // first to the service url
 				fh.logger.Debug("calling getServiceForFunction",
-					zap.String("function_name", fh.function.Name))
+					zap.String("function_name", fnMeta.Name))
 				u, err = fh.getServiceEntryFromExecutor(ctx)
 				if err != nil {
 					fh.logger.Error("error getting service url from executor",
 						zap.Error(err),
-						zap.String("function_name", fh.function.Name))
+						zap.String("function_name", fnMeta.Name))
 					return nil, err
 				}
 				// add the address in router's cache
 				fh.logger.Info("assigning service url for function",
 					zap.String("url", u.String()),
-					zap.String("function_name", fh.function.Name))
+					zap.String("function_name", fnMeta.Name))
-				fh.fmap.assign(fh.function, u)
+				fh.fmap.assign(fnMeta, u)
 			} else {
 				u, err = fh.getServiceEntryFromCache()
 				if err != nil {
@@ -583,9 +564,9 @@ func (fh *functionHandler) getServiceEntry() (serviceUrl *url.URL, serviceUrlFro
 		e := "error updating service address entry for function"
 		fh.logger.Error(e,
 			zap.Error(err),
-			zap.String("function_name", fh.function.Name),
+			zap.String("function_name", fnMeta.Name),
-			zap.String("function_namespace", fh.function.Namespace))
+			zap.String("function_namespace", fnMeta.Namespace))
-		return nil, false, errors.Wrapf(err, "%s %s_%s", e, fh.function.Name, fh.function.Namespace)
+		return nil, false, errors.Wrapf(err, "%s %s_%s", e, fnMeta.Name, fnMeta.Namespace)
 	}
 
 	record, ok := recordObj.(svcEntryRecord)
@@ -597,9 +578,9 @@ func (fh *functionHandler) getServiceEntry() (serviceUrl *url.URL, serviceUrlFro
 }
 
 // getServiceEntryFromCache returns service url entry returns from cache
-func (fh *functionHandler) getServiceEntryFromCache() (serviceUrl *url.URL, err error) {
+func (fh functionHandler) getServiceEntryFromCache() (serviceUrl *url.URL, err error) {
 	// cache lookup to get serviceUrl
-	serviceUrl, err = fh.fmap.lookup(fh.function)
+	serviceUrl, err = fh.fmap.lookup(&fh.function.Metadata)
 	if err != nil {
 		var errMsg string
 
@@ -612,7 +593,7 @@ func (fh *functionHandler) getServiceEntryFromCache() (serviceUrl *url.URL, err
 			if e.Code == ferror.ErrorNotFound {
 				return nil, nil
 			}
-			errMsg = fmt.Sprintf("Error getting function %v;s service entry from cache: %v", fh.function.Name, err)
+			errMsg = fmt.Sprintf("Error getting function %v;s service entry from cache: %v", fh.function.Metadata.Name, err)
 		}
 		return nil, ferror.MakeError(http.StatusInternalServerError, errMsg)
 	}
@@ -620,9 +601,9 @@ func (fh *functionHandler) getServiceEntryFromCache() (serviceUrl *url.URL, err
 }
 
 // getServiceEntryFromExecutor returns service url entry returns from executor
-func (fh *functionHandler) getServiceEntryFromExecutor(ctx context.Context) (*url.URL, error) {
+func (fh functionHandler) getServiceEntryFromExecutor(ctx context.Context) (*url.URL, error) {
 	// send a request to executor to specialize a new pod
-	service, err := fh.executor.GetServiceForFunction(ctx, fh.function)
+	service, err := fh.executor.GetServiceForFunction(ctx, &fh.function.Metadata)
 	if err != nil {
 		statusCode, errMsg := ferror.GetHTTPError(err)
 		fh.logger.Error("error from GetServiceForFunction",
@@ -644,3 +625,72 @@ func (fh *functionHandler) getServiceEntryFromExecutor(ctx context.Context) (*ur
 
 	return serviceUrl, nil
 }
+
+// getProxyErrorHandler returns a reverse proxy error handler
+func (fh functionHandler) getProxyErrorHandler(start time.Time, rrt *RetryingRoundTripper) func(rw http.ResponseWriter, req *http.Request, err error) {
+	return func(rw http.ResponseWriter, req *http.Request, err error) {
+		var status int
+		var msg string
+
+		switch err {
+		case context.Canceled:
+			// 499 CLIENT CLOSED REQUEST
+			// A non-standard status code introduced by nginx for the case
+			// when a client closes the connection while nginx is processing the request.
+			// Reference: https://httpstatuses.com/499
+			status = 499
+			msg = "client closes the connection"
+			fh.logger.Debug(msg, zap.Any("function", fh.function), zap.Any("request_header", req.Header))
+		case context.DeadlineExceeded:
+			status = http.StatusGatewayTimeout
+			msg = "function not responses before the timeout"
+			fh.logger.Error(msg, zap.Any("function", fh.function), zap.Any("request_header", req.Header))
+		default:
+			status = http.StatusBadGateway
+			msg = "error sending request to function"
+			fh.logger.Error(msg, zap.Error(err), zap.Any("function", fh.function), zap.Any("request_header", req.Header))
+		}
+
+		go fh.collectFunctionMetric(start, rrt, req, &http.Response{
+			StatusCode:    status,
+			ContentLength: 0,
+		})
+
+		// TODO: return error message that contains traceable UUID back to user. Issue #693
+		rw.WriteHeader(status)
+		rw.Write([]byte(msg))
+	}
+}
+
+func (fh functionHandler) collectFunctionMetric(start time.Time, rrt *RetryingRoundTripper, req *http.Request, resp *http.Response) {
+	duration := time.Since(start)
+
+	// Metrics stuff
+	funcMetricLabels := &functionLabels{
+		namespace: fh.function.Metadata.Namespace,
+		name:      fh.function.Metadata.Name,
+	}
+	httpMetricLabels := &httpLabels{
+		method: req.Method,
+	}
+	if fh.httpTrigger != nil {
+		httpMetricLabels.host = fh.httpTrigger.Spec.Host
+		httpMetricLabels.path = fh.httpTrigger.Spec.RelativeURL
+	}
+
+	// Track metrics
+	httpMetricLabels.code = resp.StatusCode
+	funcMetricLabels.cached = rrt.urlFromCache
+
+	functionCallCompleted(funcMetricLabels, httpMetricLabels,
+		duration, duration, resp.ContentLength)
+
+	// tapService before invoking roundTrip for the serviceUrl
+	if rrt.urlFromCache {
+		fh.tapService(fh.function, rrt.serviceUrl)
+	}
+
+	fh.logger.Debug("Request complete", zap.String("function", fh.function.Metadata.Name),
+		zap.Int("retry", rrt.totalRetry), zap.Duration("total-time", duration),
+		zap.Int64("content-length", resp.ContentLength))
+}

pkg/router/functionHandler_test.go

@@ -57,12 +57,12 @@ func TestFunctionProxying(t *testing.T) {
 	backendURL := createBackendService(testResponseString)
 	log.Printf("Created backend svc at %v", backendURL)
 
-	fn := &metav1.ObjectMeta{Name: "foo", Namespace: metav1.NamespaceDefault}
+	fnMeta := metav1.ObjectMeta{Name: "foo", Namespace: metav1.NamespaceDefault}
 	logger, err := zap.NewDevelopment()
 	panicIf(err)
 
 	fmap := makeFunctionServiceMap(logger, 0)
-	fmap.assign(fn, backendURL)
+	fmap.assign(&fnMeta, backendURL)
 
 	httpTrigger := &fv1.HTTPTrigger{
 		Metadata: metav1.ObjectMeta{
@@ -78,9 +78,11 @@ func TestFunctionProxying(t *testing.T) {
 	}
 
 	fh := &functionHandler{
-		logger:   logger,
+		logger: logger,
-		fmap:     fmap,
+		fmap:   fmap,
-		function: fn,
+		function: &fv1.Function{
+			Metadata: metav1.ObjectMeta{Name: "foo", Namespace: metav1.NamespaceDefault},
+		},
 		tsRoundTripperParams: &tsRoundTripperParams{
 			timeout:         50 * time.Millisecond,
 			timeoutExponent: 2,
@@ -97,7 +99,18 @@ func TestFunctionProxying(t *testing.T) {
 func TestProxyErrorHandler(t *testing.T) {
 	logger, err := zap.NewDevelopment()
 	assert.Nil(t, err)
-	errHandler := getProxyErrorHandler(logger, nil)
+
+	fh := &functionHandler{
+		logger: logger,
+		function: &fv1.Function{
+			Metadata: metav1.ObjectMeta{
+				Name:      "dummy",
+				Namespace: "dummy-bar",
+			},
+		},
+	}
+
+	errHandler := fh.getProxyErrorHandler(time.Now(), &RetryingRoundTripper{})
 
 	req, err := http.NewRequest("GET", "http://foobar.com", nil)
 	assert.Nil(t, err)

pkg/router/functionReferenceResolver.go

@@ -50,7 +50,7 @@ type (
 	// a distribution of requests across two functions.
 	resolveResult struct {
 		resolveResultType
-		functionMetadataMap        map[string]*metav1.ObjectMeta
+		functionMap                map[string]*fv1.Function
 		functionWtDistributionList []FunctionWeightDistribution
 	}
 
@@ -134,12 +134,13 @@ func (frr *functionReferenceResolver) resolveByName(namespace, name string) (*re
 	}
 
 	f := obj.(*fv1.Function)
-	functionMetadataMap := make(map[string]*metav1.ObjectMeta, 1)
+	functionMap := map[string]*fv1.Function{
-	functionMetadataMap[f.Metadata.Name] = &f.Metadata
+		f.Metadata.Name: f,
+	}
 
 	rr := resolveResult{
-		resolveResultType:   resolveResultSingleFunction,
+		resolveResultType: resolveResultSingleFunction,
-		functionMetadataMap: functionMetadataMap,
+		functionMap:       functionMap,
 	}
 
 	return &rr, nil
@@ -147,7 +148,7 @@ func (frr *functionReferenceResolver) resolveByName(namespace, name string) (*re
 
 func (frr *functionReferenceResolver) resolveByFunctionWeights(namespace string, fr *fv1.FunctionReference) (*resolveResult, error) {
 
-	functionMetadataMap := make(map[string]*metav1.ObjectMeta)
+	functionMap := make(map[string]*fv1.Function)
 	fnWtDistrList := make([]FunctionWeightDistribution, 0)
 	sumPrefix := 0
 
@@ -167,19 +168,18 @@ func (frr *functionReferenceResolver) resolveByFunctionWeights(namespace string,
 		}
 
 		f := obj.(*fv1.Function)
-		functionMetadataMap[f.Metadata.Name] = &f.Metadata
+		functionMap[f.Metadata.Name] = f
 		sumPrefix = sumPrefix + functionWeight
 		fnWtDistrList = append(fnWtDistrList, FunctionWeightDistribution{
 			name:      functionName,
 			weight:    functionWeight,
 			sumPrefix: sumPrefix,
 		})
-
 	}
 
 	rr := resolveResult{
 		resolveResultType:          resolveResultMultipleFunctions,
-		functionMetadataMap:        functionMetadataMap,
+		functionMap:                functionMap,
 		functionWtDistributionList: fnWtDistrList,
 	}
 

pkg/router/httpTriggers.go

@@ -142,7 +142,7 @@ func (ts *HTTPTriggerSet) getRouter(fnTimeoutMap map[types.UID]int) *mux.Router
 			fmap:                     ts.functionServiceMap,
 			executor:                 ts.executor,
 			httpTrigger:              &trigger,
-			functionMetadataMap:      rr.functionMetadataMap,
+			functionMap:              rr.functionMap,
 			fnWeightDistributionList: rr.functionWtDistributionList,
 			tsRoundTripperParams:     ts.tsRoundTripperParams,
 			isDebugEnv:               ts.isDebugEnv,
@@ -158,8 +158,8 @@ func (ts *HTTPTriggerSet) getRouter(fnTimeoutMap map[types.UID]int) *mux.Router
 		// deployment. For more details, please check "handler" function of functionHandler.
 
 		if rr.resolveResultType == resolveResultSingleFunction {
-			for _, metadata := range fh.functionMetadataMap {
+			for _, fn := range fh.functionMap {
-				fh.function = metadata
+				fh.function = fn
 			}
 		}
 
@@ -185,20 +185,19 @@ func (ts *HTTPTriggerSet) getRouter(fnTimeoutMap map[types.UID]int) *mux.Router
 
 	// Internal triggers for each function by name. Non-http
 	// triggers route into these.
-	for _, function := range ts.functions {
+	for i := range ts.functions {
-		m := function.Metadata
+		fn := ts.functions[i]
-
 		fh := &functionHandler{
-			logger:                 ts.logger.Named(m.Name),
+			logger:                 ts.logger.Named(fn.Metadata.Name),
 			fmap:                   ts.functionServiceMap,
-			function:               &m,
+			function:               &fn,
 			executor:               ts.executor,
 			tsRoundTripperParams:   ts.tsRoundTripperParams,
 			isDebugEnv:             ts.isDebugEnv,
 			svcAddrUpdateThrottler: ts.svcAddrUpdateThrottler,
 			functionTimeoutMap:     fnTimeoutMap,
 		}
-		muxRouter.HandleFunc(utils.UrlForFunction(function.Metadata.Name, function.Metadata.Namespace), fh.handler)
+		muxRouter.HandleFunc(utils.UrlForFunction(fn.Metadata.Name, fn.Metadata.Namespace), fh.handler)
 	}
 
 	// Healthz endpoint for the router.
@@ -263,8 +262,8 @@ func (ts *HTTPTriggerSet) initFunctionController() (k8sCache.Store, k8sCache.Con
 				// update resolver function reference cache
 				for key, rr := range ts.resolver.copy() {
 					if key.namespace == fn.Metadata.Namespace &&
-						rr.functionMetadataMap[fn.Metadata.Name] != nil &&
+						rr.functionMap[fn.Metadata.Name] != nil &&
-						rr.functionMetadataMap[fn.Metadata.Name].ResourceVersion != fn.Metadata.ResourceVersion {
+						rr.functionMap[fn.Metadata.Name].Metadata.ResourceVersion != fn.Metadata.ResourceVersion {
 						// invalidate resolver cache
 						ts.logger.Debug("invalidating resolver cache")
 						err := ts.resolver.delete(key.namespace, key.triggerName, key.triggerResourceVersion)

pkg/router/ingress.go

@@ -71,6 +71,9 @@ func deleteIngress(logger *zap.Logger, trigger *fv1.HTTPTrigger, kubeClient *kub
 }
 
 func updateIngress(logger *zap.Logger, oldT *fv1.HTTPTrigger, newT *fv1.HTTPTrigger, kubeClient *kubernetes.Clientset) {
+	if !oldT.Spec.CreateIngress && !newT.Spec.CreateIngress {
+		return
+	}
 
 	if !oldT.Spec.CreateIngress && newT.Spec.CreateIngress {
 		createIngress(logger, newT, kubeClient)

pkg/router/router_test.go

@@ -32,12 +32,12 @@ import (
 
 func TestRouter(t *testing.T) {
 	// metadata for a fake function
-	fn := &metav1.ObjectMeta{Name: "foo", Namespace: metav1.NamespaceDefault}
+	fnMeta := metav1.ObjectMeta{Name: "foo", Namespace: metav1.NamespaceDefault}
 
 	// and a reference to it
 	fr := fv1.FunctionReference{
 		Type: types.FunctionReferenceTypeFunctionName,
-		Name: fn.Name,
+		Name: fnMeta.Name,
 	}
 
 	// start a fake service
@@ -49,7 +49,7 @@ func TestRouter(t *testing.T) {
 
 	// set up the cache with this fake service
 	fmap := makeFunctionServiceMap(logger, 0)
-	fmap.assign(fn, testServiceUrl)
+	fmap.assign(&fnMeta, testServiceUrl)
 
 	// HTTP trigger set with a trigger for this function
 	triggers, _, _ := makeHTTPTriggerSet(logger, fmap, nil, nil, nil, nil,
@@ -81,12 +81,14 @@ func TestRouter(t *testing.T) {
 		triggerResourceVersion: "1234",
 	}
 
-	fnMetaMap := make(map[string]*metav1.ObjectMeta, 1)
+	fnMetaMap := make(map[string]*fv1.Function, 1)
-	fnMetaMap[fn.Name] = fn
+	fnMetaMap[fnMeta.Name] = &fv1.Function{
+		Metadata: fnMeta,
+	}
 
 	rr := resolveResult{
-		resolveResultType:   resolveResultSingleFunction,
+		resolveResultType: resolveResultSingleFunction,
-		functionMetadataMap: fnMetaMap,
+		functionMap:       fnMetaMap,
 	}
 	frr.refCache.Set(nfr, rr)
 

pkg/types/types.go

@@ -91,7 +91,6 @@ const (
 )
 
 const EXECUTOR_INSTANCEID_LABEL = fv1.EXECUTOR_INSTANCEID_LABEL
-const POOLMGR_INSTANCEID_LABEL = fv1.POOLMGR_INSTANCEID_LABEL
 
 const (
 	ChecksumTypeSHA256 = fv1.ChecksumTypeSHA256
@@ -121,22 +120,18 @@ const (
 
 // executor kubernetes object label key
 const (
-	ENVIRONMENT_NAMESPACE = "environmentNamespace"
+	ENVIRONMENT_NAMESPACE     = "environmentNamespace"
-	ENVIRONMENT_NAME      = "environmentName"
+	ENVIRONMENT_NAME          = "environmentName"
-	ENVIRONMENT_UID       = "environmentUid"
+	ENVIRONMENT_UID           = "environmentUid"
-	FUNCTION_NAMESPACE    = "functionNamespace"
+	FUNCTION_NAMESPACE        = "functionNamespace"
-	FUNCTION_NAME         = "functionName"
+	FUNCTION_NAME             = "functionName"
-	FUNCTION_UID          = "functionUid"
+	FUNCTION_UID              = "functionUid"
-	EXECUTOR_TYPE         = "executorType"
+	FUNCTION_RESOURCE_VERSION = "functionResourceVersion"
+	EXECUTOR_TYPE             = "executorType"
 )
 
 const (
-	ExecutorTypePoolmgr   = fv1.ExecutorTypePoolmgr
+	ANNOTATION_SVC_HOST = "svcHost"
-	ExecutorTypeNewdeploy = fv1.ExecutorTypeNewdeploy
-)
-
-const (
-	StrategyTypeExecution = fv1.StrategyTypeExecution
 )
 
 const (

pkg/utils/utils.go

@@ -17,12 +17,14 @@ limitations under the License.
 package utils
 
 import (
+	"context"
 	"crypto/sha256"
 	"encoding/hex"
 	"fmt"
 	"io"
 	"io/ioutil"
 	"net"
+	"net/http"
 	"os"
 	"path/filepath"
 	"strings"
@@ -30,10 +32,12 @@ import (
 	"github.com/mholt/archiver"
 	"github.com/pkg/errors"
 	uuid "github.com/satori/go.uuid"
+	"golang.org/x/net/context/ctxhttp"
 	apiv1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
 	fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
+	"github.com/fission/fission/pkg/fission-cli/console"
 )
 
 func UrlForFunction(name, namespace string) string {
@@ -99,21 +103,34 @@ func GetTempDir() (string, error) {
 	return dir, err
 }
 
-// FindAllGlobs returns a list of globs of input list.
+// FindAllGlobs ignores all hidden files and returns a list of globs of input list.
-func FindAllGlobs(inputList []string) ([]string, error) {
+func FindAllGlobs(paths ...string) ([]string, error) {
 	files := make([]string, 0)
-	for _, glob := range inputList {
+	for _, p := range paths {
-		f, err := filepath.Glob(glob)
+		// use absolute path to find files
+		path, err := filepath.Abs(p)
 		if err != nil {
-			return nil, errors.Errorf("invalid glob %v: %v", glob, err)
+			return nil, errors.Wrapf(err, "error getting absolute path of path '%v'", p)
+		}
+		globs, err := filepath.Glob(path)
+		if err != nil {
+			return nil, errors.Errorf("invalid glob %v: %v", path, err)
+		}
+		for _, f := range globs {
+			// ignore hidden file.
+			if strings.HasPrefix(filepath.Base(f), ".") {
+				console.Verbose(2, "Ignore hidden file '%v'", f)
+				continue
+			}
+			files = append(files, f)
+			// xxx handle excludeGlobs here
 		}
-		files = append(files, f...)
 	}
 	return files, nil
 }
 
 func MakeZipArchive(targetName string, globs ...string) (string, error) {
-	files, err := FindAllGlobs(globs)
+	files, err := FindAllGlobs(globs...)
 	if err != nil {
 		return "", err
 	}
@@ -198,3 +215,35 @@ func GetChecksum(src io.Reader) (*fv1.Checksum, error) {
 func IsURL(str string) bool {
 	return strings.HasPrefix(str, "http://") || strings.HasPrefix(str, "https://")
 }
+
+func DownloadUrl(ctx context.Context, httpClient *http.Client, url string, localPath string) error {
+	resp, err := ctxhttp.Get(ctx, httpClient, url)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+
+	w, err := os.Create(localPath)
+	if err != nil {
+		return err
+	}
+	defer w.Close()
+
+	_, err = io.Copy(w, resp.Body)
+	if err != nil {
+		return err
+	}
+
+	// flushing write buffer to file
+	err = w.Sync()
+	if err != nil {
+		return err
+	}
+
+	err = os.Chmod(localPath, 0600)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}

test/test_utils.sh

@@ -512,6 +512,7 @@ run_all_tests() {
     export JOBS=6
     $ROOT/test/run_test.sh \
         $ROOT/test/tests/test_canary.sh \
+        $ROOT/test/tests/test_fn_update/test_idle_objects_reaper.sh \
         $ROOT/test/tests/mqtrigger/kafka/test_kafka.sh \
         $ROOT/test/tests/test_annotations.sh \
         $ROOT/test/tests/test_archive_pruner.sh \
@@ -519,7 +520,6 @@ run_all_tests() {
         $ROOT/test/tests/test_buildermgr.sh \
         $ROOT/test/tests/test_env_vars.sh \
         $ROOT/test/tests/test_environments/test_python_env.sh \
-        $ROOT/test/tests/test_fn_update/test_idle_objects_reaper.sh \
         $ROOT/test/tests/test_function_test/test_fn_test.sh \
         $ROOT/test/tests/test_function_update.sh \
         $ROOT/test/tests/test_ingress.sh \
@@ -527,19 +527,22 @@ run_all_tests() {
         $ROOT/test/tests/test_logging/test_function_logs.sh \
         $ROOT/test/tests/test_node_hello_http.sh \
         $ROOT/test/tests/test_package_command.sh \
+        $ROOT/test/tests/test_package_checksum.sh \
         $ROOT/test/tests/test_pass.sh \
         $ROOT/test/tests/test_router_cache_invalidation.sh \
         $ROOT/test/tests/test_specs/test_spec.sh \
         $ROOT/test/tests/test_specs/test_spec_multifile.sh \
+        $ROOT/test/tests/test_specs/test_ignore_hidden_file.sh \
         $ROOT/test/tests/test_specs/test_spec_merge/test_spec_merge.sh \
+        $ROOT/test/tests/test_specs/test_spec_archive/test_spec_archive.sh \
         $ROOT/test/tests/test_environments/test_tensorflow_serving_env.sh \
         $ROOT/test/tests/test_environments/test_go_env.sh \
         $ROOT/test/tests/mqtrigger/nats/test_mqtrigger.sh \
-        $ROOT/test/tests/mqtrigger/nats/test_mqtrigger_error.sh
+        $ROOT/test/tests/mqtrigger/nats/test_mqtrigger_error.sh \
+        $ROOT/test/tests/test_huge_response/test_huge_response.sh
     FAILURES=$?
 
-    # FIXME: run tests with newdeploy one by one.
+    export JOBS=3
-    export JOBS=2
     $ROOT/test/run_test.sh \
         $ROOT/test/tests/test_backend_newdeploy.sh \
         $ROOT/test/tests/test_environments/test_java_builder.sh \

test/tests/test_fn_update/test_idle_objects_reaper.sh

@@ -41,7 +41,7 @@ timeout 60 bash -c "test_fn ${fn}-gpm 'world'"
 log "Waiting for idle pod reaper to recycle resources"
 # the LIST_OLD function list fsvc older than 2 mins
 # so in worst case, we need to wait for up to 4 mins + some buffer
-sleep 260
+sleep 300
 
 # The replicas of function deployment should be 0 due to minScale = 0
 ndDeployReplicas=$(kubectl -n $FUNCTION_NAMESPACE get deploy -l functionName=${fn}-nd -ojsonpath='{.items[0].spec.replicas}')

test/tests/test_huge_response/hello.go

@@ -0,0 +1,16 @@
+package main
+
+import (
+	"io/ioutil"
+	"net/http"
+)
+
+// Handler is the entry point for this fission function
+func Handler(w http.ResponseWriter, r *http.Request) {
+	bytes, err := ioutil.ReadAll(r.Body)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusInternalServerError)
+		return
+	}
+	w.Write(bytes)
+}

test/tests/test_huge_response/test_huge_response.sh

@@ -0,0 +1,73 @@
+#!/bin/bash
+
+set -euo pipefail
+source $(dirname $0)/../../utils.sh
+
+TEST_ID=$(generate_test_id)
+echo "TEST_ID = $TEST_ID"
+
+ROOT=$(dirname $0)/../../..
+
+pushd $ROOT/test/tests/test_huge_response
+
+cleanup() {
+    clean_resource_by_id $TEST_ID
+    rm -rf response.json
+    popd
+}
+
+retryPost() {
+    local fn=$1
+    log "Send huge JSON request body"
+
+    set +e
+    while true; do
+        curl -X POST http://$FISSION_ROUTER/$fn \
+            -H "Content-Type: application/json" \
+            --data-binary "@generated.json" > response.json
+
+        difftext=$(diff generated.json response.json)
+        if [ -z "$difftext" ]; then
+            break
+        else
+            echo "Receive truncated body"
+            sleep 1
+        fi
+    done
+    set -e
+}
+export -f retryPost
+
+if [ -z "${TEST_NOCLEANUP:-}" ]; then
+    trap cleanup EXIT
+else
+    log "TEST_NOCLEANUP is set; not cleaning up test artifacts afterwards."
+fi
+
+env=go-$TEST_ID
+fn_poolmgr=hello-go-poolmgr-$TEST_ID
+
+log "Creating environment for Golang"
+fission env create --name $env --image $GO_RUNTIME_IMAGE --builder $GO_BUILDER_IMAGE --period 5
+
+timeout 90 bash -c "wait_for_builder $env"
+
+pkgName=$(generate_test_id)
+fission package create --name $pkgName --src hello.go --env $env
+
+# wait for build to finish at most 90s
+timeout 90 bash -c "waitBuild $pkgName"
+
+log "Creating function for Golang"
+fission fn create --name $fn_poolmgr --env $env --pkg $pkgName --entrypoint Handler
+
+log "Creating route for function"
+fission route create --name $fn_poolmgr --function $fn_poolmgr --url /$fn_poolmgr --method POST
+
+log "Waiting for router & pools to catch up"
+sleep 5
+
+log "Testing function"
+timeout 20 bash -c "retryPost $fn_poolmgr"
+
+log "Test PASSED"

test/tests/test_package_checksum.sh

@@ -0,0 +1,112 @@
+#!/bin/bash
+
+set -euo pipefail
+source $(dirname $0)/../utils.sh
+
+TEST_ID=$(generate_test_id)
+echo "TEST_ID = $TEST_ID"
+
+tmp_dir="/tmp/test-$TEST_ID"
+mkdir -p $tmp_dir
+
+ROOT=$(dirname $0)/../..
+
+checkpkgsum() {
+    local pkg=$1
+    local sum=$2
+
+    pkgsum=$(kubectl -n default get packages ${pkg} -o jsonpath='{.spec.deployment.checksum.sum}')
+
+    if [ "${sum}" != "${pkgsum}" ]; then
+        log "have different sha256 checksum: ${sum} vs. ${pkgsum}"
+        kubectl -n default get packages ${pkg} -o yaml
+        exit 1
+    fi
+}
+
+cleanup() {
+    log "Cleaning up..."
+    clean_resource_by_id $TEST_ID
+    rm -rf $tmp_dir
+}
+
+if [ -z "${TEST_NOCLEANUP:-}" ]; then
+    trap cleanup EXIT
+else
+    log "TEST_NOCLEANUP is set; not cleaning up test artifacts afterwards."
+fi
+
+log "Download test script"
+
+url1="https://raw.githubusercontent.com/fission/fission/master/examples/nodejs/hello.js"
+url2="https://raw.githubusercontent.com/fission/fission/master/examples/nodejs/hello-callback.js"
+
+wget ${url1}
+sum=$(shasum -a 256 hello.js|cut -d' ' -f 1)
+
+wget ${url2}
+sum2=$(shasum -a 256 hello-callback.js|cut -d' ' -f 1)
+
+env="nodejs-${TEST_ID}"
+
+log "Function with file URL"
+fn1="fn1-$TEST_ID"
+fission env create --name ${env} --image $NODE_RUNTIME_IMAGE --period 5
+fission fn create --name ${fn1} --env ${env} --code ${url1}
+pkgname=$(kubectl -n default get functions ${fn1} -o jsonpath="{.spec.package.packageref.name}")
+checkpkgsum ${pkgname} ${sum}
+
+log "Creating route"
+fission route create --name ${fn1} --function ${fn1} --url /${fn1} --method GET
+sleep 3
+
+timeout 60 bash -c "test_fn ${fn1} 'hello, world'"
+
+log "Update function with file URL"
+fission fn update --name ${fn1} --env ${env} --code ${url2}
+pkgname=$(kubectl -n default get functions ${fn1} -o jsonpath="{.spec.package.packageref.name}")
+checkpkgsum ${pkgname} ${sum2}
+
+sleep 3
+
+timeout 60 bash -c "test_fn ${fn1} 'Hello, world callback!'"
+
+log "Function with file URL & checksum"
+fn2="fn2-$TEST_ID"
+fission fn create --name ${fn2} --env ${env} --code ${url1} --deploychecksum ${sum}
+pkgname=$(kubectl -n default get functions ${fn2} -o jsonpath="{.spec.package.packageref.name}")
+checkpkgsum ${pkgname} ${sum}
+
+log "Creating route"
+fission route create --name ${fn2} --function ${fn2} --url /${fn2} --method GET
+
+timeout 60 bash -c "test_fn ${fn2} 'hello, world'"
+
+log "Function with file URL & insecure"
+fn3="fn3-$TEST_ID"
+fission fn create --name ${fn3} --env ${env} --code ${url1} --insecure
+pkgname=$(kubectl -n default get functions ${fn3} -o jsonpath="{.spec.package.packageref.name}")
+checkpkgsum ${pkgname} ""
+
+log "Creating route"
+fission route create --name ${fn3} --function ${fn3} --url /${fn3} --method GET
+sleep 3
+
+timeout 60 bash -c "test_fn ${fn3} 'hello, world'"
+
+pkg1="pkg1-$TEST_ID"
+pkg2="pkg2-$TEST_ID"
+pkg3="pkg3-$TEST_ID"
+
+fission pkg create --name ${pkg1} --env ${env} --code ${url1}
+checkpkgsum ${pkg1} ${sum}
+fission pkg update --name ${pkg1} --env ${env} --code ${url2}
+checkpkgsum ${pkg1} ${sum2}
+fission pkg create --name ${pkg2} --env ${env} --code ${url1} --deploychecksum ${sum}
+checkpkgsum ${pkg2} ${sum}
+fission pkg create --name ${pkg3} --env ${env} --code ${url1} --insecure
+checkpkgsum ${pkg3} ""
+
+log "Test PASSED"
+
+exit 0

test/tests/test_specs/test_ignore_hidden_file.sh

@@ -0,0 +1,65 @@
+#!/bin/bash
+
+set -euo pipefail
+source $(dirname $0)/../../utils.sh
+ROOT=` realpath $(dirname $0)/../../../`
+TEST_ID=$(generate_test_id)
+
+cleanup() {
+    log "Cleaning up..."
+    fission spec destroy || true
+    rm -rf document specs
+    rm -rf ${TEST_ID}
+    popd
+}
+
+if [ -z "${TEST_NOCLEANUP:-}" ]; then
+    trap cleanup EXIT
+else
+    log "TEST_NOCLEANUP is set; not cleaning up test artifacts afterwards."
+fi
+
+tmp_dir="/tmp/test-$TEST_ID"
+mkdir -p $tmp_dir
+
+pushd $tmp_dir
+
+mkdir -p document
+cp $ROOT/examples/nodejs/hello.js document/h1.js
+cp $ROOT/examples/nodejs/hello.js document/h2.js
+
+log "Create hidden file"
+touch document/.im_invisible
+
+log "Create specs"
+fission spec init
+fission pkg list
+
+#fission env create --name nodejs --image fission/node-env --period 5 --version 2 --spec
+fission pkg create --name nodejs --env nodejs --deploy "document/*" --spec
+
+log "Apply specs"
+fission --verbosity 2 spec apply
+
+mkdir ${TEST_ID}
+fission pkg getdeploy --name nodejs > ${TEST_ID}/a.zip
+unzip ${TEST_ID}/a.zip -d ${TEST_ID}/
+
+log "Check whether hidden file exists"
+if [ -f ${TEST_ID}/.im_invisible ];
+then
+    log "Found hidden file"
+    ls -al ${TEST_ID}
+    exit 1
+fi
+
+log "Check file amount"
+fileamount=$(ls -al ${TEST_ID} | grep -v total | wc -l)
+if [ ! ${fileamount} -eq 5 ];
+then
+  log "File amount incorrect, expect 5"
+  ls -al ${TEST_ID}
+  exit 1
+fi
+
+log "Test PASSED"

test/tests/test_specs/test_spec_archive/specs/README

@@ -0,0 +1,42 @@
+
+Fission Specs
+=============
+
+This is a set of specifications for a Fission app.  This includes functions,
+environments, and triggers; we collectively call these things "resources".
+
+How to use these specs
+----------------------
+
+These specs are handled with the 'fission spec' command.  See 'fission spec --help'.
+
+'fission spec apply' will "apply" all resources specified in this directory to your
+cluster.  That means it checks what resources exist on your cluster, what resources are
+specified in the specs directory, and reconciles the difference by creating, updating or
+deleting resources on the cluster.
+
+'fission spec apply' will also package up your source code (or compiled binaries) and
+upload the archives to the cluster if needed.  It uses 'ArchiveUploadSpec' resources in
+this directory to figure out which files to archive.
+
+You can use 'fission spec apply --watch' to watch for file changes and continuously keep
+the cluster updated.
+
+You can add YAMLs to this directory by writing them manually, but it's easier to generate
+them.  Use 'fission function create --spec' to generate a function spec,
+'fission environment create --spec' to generate an environment spec, and so on.
+
+You can edit any of the files in this directory, except 'fission-deployment-config.yaml',
+which contains a UID that you should never change.  To apply your changes simply use
+'fission spec apply'.
+
+fission-deployment-config.yaml
+------------------------------
+
+fission-deployment-config.yaml contains a UID.  This UID is what fission uses to correlate
+resources on the cluster to resources in this directory.
+
+All resources created by 'fission spec apply' are annotated with this UID.  Resources on
+the cluster that are _not_ annotated with this UID are never modified or deleted by
+fission.
+

test/tests/test_specs/test_spec_archive/specs/env-node.yaml

@@ -0,0 +1,16 @@
+apiVersion: fission.io/v1
+kind: Environment
+metadata:
+  creationTimestamp: null
+  name: dummyfoobarnode
+  namespace: default
+spec:
+  builder:
+    command: build
+    image: fission/node-builder:1.6.0
+  keeparchive: false
+  poolsize: 3
+  runtime:
+    image: fission/node-env:1.6.0
+  terminationGracePeriod: 20
+  version: 2

test/tests/test_specs/test_spec_archive/specs/fission-deployment-config.yaml

@@ -0,0 +1,7 @@
+# This file is generated by the 'fission spec init' command.
+# See the README in this directory for background and usage information.
+# Do not edit the UID below: that will break 'fission spec apply'
+apiVersion: fission.io/v1
+kind: DeploymentConfig
+name: test-spec-archive
+uid: 04b21526-8873-4dc2-b897-e87ed5347670

test/tests/test_specs/test_spec_archive/specs/functions.yaml

@@ -0,0 +1,56 @@
+apiVersion: fission.io/v1
+kind: Function
+metadata:
+  creationTimestamp: null
+  name: sourcearchive
+  namespace: default
+spec:
+  InvokeStrategy:
+    ExecutionStrategy:
+      ExecutorType: poolmgr
+      MaxScale: 0
+      MinScale: 0
+      SpecializationTimeout: 0
+      TargetCPUPercent: 0
+    StrategyType: execution
+  configmaps: null
+  environment:
+    name: dummyfoobarnode
+    namespace: default
+  functionTimeout: 60
+  package:
+    functionName: source
+    packageref:
+      name: sourcearchive
+      namespace: default
+  resources: {}
+  secrets: null
+
+---
+apiVersion: fission.io/v1
+kind: Function
+metadata:
+  creationTimestamp: null
+  name: deployarchive
+  namespace: default
+spec:
+  InvokeStrategy:
+    ExecutionStrategy:
+      ExecutorType: poolmgr
+      MaxScale: 0
+      MinScale: 0
+      SpecializationTimeout: 0
+      TargetCPUPercent: 0
+    StrategyType: execution
+  configmaps: null
+  environment:
+    name: dummyfoobarnode
+    namespace: default
+  functionTimeout: 60
+  package:
+    functionName: deploy
+    packageref:
+      name: deployarchive
+      namespace: default
+  resources: {}
+  secrets: null

test/tests/test_specs/test_spec_archive/specs/pkg-document-deploy-archive.yaml

@@ -0,0 +1,24 @@
+include:
+  - func/*
+kind: ArchiveUploadSpec
+name: functions-deploy-archive
+
+---
+apiVersion: fission.io/v1
+kind: Package
+metadata:
+  creationTimestamp: null
+  name: deployarchive
+  namespace: default
+spec:
+  deployment:
+    checksum: {}
+    type: url
+    url: archive://functions-source-archive
+  environment:
+    name: dummyfoobarnode
+    namespace: default
+  source:
+    checksum: {}
+status:
+  buildstatus: none

test/tests/test_specs/test_spec_archive/specs/pkg-document-source-archive.yaml

@@ -0,0 +1,24 @@
+include:
+  - func/*
+kind: ArchiveUploadSpec
+name: functions-source-archive
+
+---
+apiVersion: fission.io/v1
+kind: Package
+metadata:
+  creationTimestamp: null
+  name: sourcearchive
+  namespace: default
+spec:
+  deployment:
+    checksum: {}
+  environment:
+    name: dummyfoobarnode
+    namespace: default
+  source:
+    checksum: {}
+    type: url
+    url: archive://functions-deploy-archive
+status:
+  buildstatus: pending

test/tests/test_specs/test_spec_archive/test_spec_archive.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+set -euo pipefail
+source $(dirname $0)/../../../utils.sh
+ROOT=` realpath $(dirname $0)/../../../../`
+
+cleanup() {
+    log "Cleaning up..."
+    fission spec destroy
+    rm -rf func
+    popd
+}
+
+if [ -z "${TEST_NOCLEANUP:-}" ]; then
+    trap cleanup EXIT
+else
+    log "TEST_NOCLEANUP is set; not cleaning up test artifacts afterwards."
+fi
+
+pushd $(dirname $0)
+
+[ -d specs ]
+[ -f specs/README ]
+[ -f specs/fission-deployment-config.yaml ]
+
+mkdir -p func
+cp $ROOT/examples/nodejs/hello.js func/deploy.js
+cp $ROOT/examples/nodejs/hello.js func/source.js
+
+fission spec destroy || true
+
+log "Apply specs"
+fission --verbosity 2 spec apply
+
+log "verify deployarchive function works"
+fission fn test --name deployarchive
+
+timeout 60s bash -c "waitBuild sourcearchive"
+
+log "verify sourcearchive function works"
+fission fn test --name sourcearchive
+
+log "Test PASSED"

test/tests/test_specs/test_spec_merge/test_spec_merge.sh

@@ -20,7 +20,6 @@ pushd $(dirname $0)
 fission spec apply
 
 fission fn test --name $fn_p
-
 fission fn test --name $fn_nd
 
 hnd=$(kubectl -n $FUNCTION_NAMESPACE get deployment -l=functionName=$fn_nd -ojsonpath='{.items[0].spec.template.spec.hostname}')

test/utils.sh

@@ -44,6 +44,10 @@ clean_resource_by_id() {
 }
 
 test_fn() {
+    if [ -z $FISSION_ROUTER ]; then
+        log "Environment FISSION_ROUTER not set"
+        exit 1
+    fi
     url="http://$FISSION_ROUTER/$1"
     expect=$2
     test_response $url $expect
@@ -162,8 +166,8 @@ export FISSION_NATS_STREAMING_URL="http://defaultFissionAuthToken@$(kubectl -n $
 ## To change the environment image setting for CI test, please refer run_all_tests() in test_utils.sh.
 export PYTHON_RUNTIME_IMAGE=${PYTHON_RUNTIME_IMAGE:-fission/python-env}
 export PYTHON_BUILDER_IMAGE=${PYTHON_BUILDER_IMAGE:-fission/python-builder}
-export GO_RUNTIME_IMAGE=${GO_RUNTIME_IMAGE:-fission/go-env}
+export GO_RUNTIME_IMAGE=${GO_RUNTIME_IMAGE:-fission/go-env-1.12}
-export GO_BUILDER_IMAGE=${GO_BUILDER_IMAGE:-fission/go-builder}
+export GO_BUILDER_IMAGE=${GO_BUILDER_IMAGE:-fission/go-builder-1.12}
 export JVM_RUNTIME_IMAGE=${JVM_RUNTIME_IMAGE:-fission/jvm-env}
 export JVM_BUILDER_IMAGE=${JVM_BUILDER_IMAGE:-fission/jvm-builder}
 export NODE_RUNTIME_IMAGE=${NODE_RUNTIME_IMAGE:-fission/node-env}