Compare commits

..
21 Commits
Author SHA1 Message Date
Ta-Ching Chen d9a0ee1e33 Fission 1.7.0-rc.2 2019-11-28 00:40:47 +08:00
Ta-Ching ChenandGitHub 5cfc8dee5e Push extra tag to fit go module semver tag format (#1444) 2019-11-28 00:40:20 +08:00
Ta-Ching ChenandGitHub 1ad7ac2dcf Adopt existing orphan kubernetes resources when executor starts up (#1443)
Previously, once the executor is deleted for reasons (like upgrade or cluster scale-in),
the new executor deletes all existing resources created by the old executor and creates
new one. This mechanism becomes a problem when there are requests connecting to the
existing pods. Also in the worst case, the cluster may not have enough resources to create
new pods and cause service downtime.

This PR let each executor type adopts existing resources before starting the executor
API services, and so the alive connections won't experience failure. However, the requests
send to the function that doesn't have alive function pods will still fail due to the
executor is in bootstrapping.
2019-11-27 23:08:45 +08:00
Ta-Ching ChenandGitHub 86446c8879 Revert "Try to fix flaky canary test (#1441)" (#1442)
This reverts commit df857e3194.
Looks like there are some problems when running the test against GKE, so revert it.
2019-11-27 16:19:33 +08:00
Ta-Ching ChenandGitHub df857e3194 Try to fix flaky canary test (#1441) 2019-11-27 02:57:22 +08:00
Ta-Ching ChenandGitHub ad7a3951c5 Fix router tries to update ingress when createIngress is false (#1440) 2019-11-27 00:28:48 +08:00
Ta-Ching ChenandGitHub 7e8e968013 Fix poolmanager sets 0 timeout for function specialization (#1439) 2019-11-26 19:22:49 +08:00
Ta-Ching ChenandGitHub e38baeec36 Add huge response body test (#1437) 2019-11-26 12:50:41 +08:00
Ta-Ching ChenandGitHub ca28f962d4 Return error when specialization failed (#1436) 2019-11-26 09:23:06 +08:00
Ta-Ching ChenandGitHub dfb2c073d2 Collect function metrics after finishing request (#1433) 2019-11-26 02:33:40 +08:00
Ta-Ching ChenandGitHub 51b264e8ca Fix poolmanager terminates running function pod periodically (#1435)
The pool manager keeps terminating function pod periodically even there are
traffic to the function. The root cause is that executor, poolmgr, newdeploy
manage their own functionServiceCache separately. And when router taps a
function, executor updates the access time of the function service entry in its
own cache without notifying executor types to do the update as well. Hence,
the access time of function service entry in poolmanager cache never gets updated.
Due to the access time never gets updated, the idle pod reaper in poolmanager
then thinks the function pod is in idle state and recycle it.

This PR removes the cache in executor itself, and when router tries to tap a function,
executor will call executor type to tap the function and update access time.
2019-11-26 01:16:30 +08:00
Ta-Ching ChenandGitHub 6d2fe08973 Allow to tap multiple function services at one time (#1434)
The router taps function service one by one which is inefficient and
increases the burden of executor. This PR aggregates all requests into
one to solve the problem mentioned above.
2019-11-25 18:44:48 +08:00
Ta-Ching ChenandGitHub ccc551112b Fix poolmanager crashes when failed to list environment (#1432) 2019-11-24 05:59:38 +08:00
Ta-Ching ChenandGitHub 1a5537f4ba Ability to pull builder image from private registry (#1431) 2019-11-24 05:17:56 +08:00
Ta-Ching ChenandGitHub 5c2f0c5f4a Add checksum and insecure flag for user to skip checksum generation (#1430) 2019-11-24 01:35:01 +08:00
Ta-Ching ChenandGitHub a66de4c601 Support to set imagePullSecret when creating environment (#1429) 2019-11-22 18:19:57 +08:00
Ta-Ching ChenandGitHub af73d0ce1a Fix no kubeobjs get created if fn created before env creation (#1428)
When a function is created before the creation of the environment it's used, the newdeploy will not be able to create kube objs. Hence no function service record is inserted into the cache.

When getFuncSvc is called, the newdeploy tries to find the record in service cache in order to create kube objs with the same name used in previous kubeobjs creation. However, due to no record in the cache, a NotFound error is returned directly and causes the problem. To solve this, we use fn meta UID to ensure we always get the same obj name instead of getting it from the cache.
2019-11-22 01:37:02 +08:00
Ta-Ching ChenandGitHub 04654ca465 Improve compatibility with Openshift (#1424) 2019-11-21 16:56:53 +08:00
Ta-Ching ChenandGitHub af10579be9 Fix verbosity flag not show in usage (#1425) 2019-11-21 12:36:50 +08:00
Ta-Ching ChenandGitHub 30d24a85b5 Fix truncated body returned from router (#1420)
If the context of request is closed before ReverseProxy finishing writing
a huge response body to the response writer, the client will only receive
a truncated response body.

To solve this, move the context cancel after ReverseProxy finished.
2019-11-21 01:04:35 +08:00
Ta-Ching Chen d8b386a812 Update CHANGELOG for 1.7.0-rc.1 2019-11-18 10:32:46 +00:00
65 changed files with 10068 additions and 996 deletions
+88 -23
View File
@@ -1,35 +1,100 @@
# 1.6.0
# 1.7.0-rc.1
[Documentation](https://docs.fission.io/)
## Downloads for 1.6.0
## Downloads for 1.7.0-rc.1
filename | sha256 hash
-------- | -----------
[fission-core-1.6.0-minikube.yaml](https://github.com/fission/fission/releases/download/1.6.0/fission-core-1.6.0-minikube.yaml) | `a9786ff2b73f2c3b67337278707c7fb2d55bcee40524dc38bc9e3ec48410fa8b`
[fission-all-1.6.0.yaml](https://github.com/fission/fission/releases/download/1.6.0/fission-all-1.6.0.yaml) | `ef1c69a69d565a07f82e52936e256f7515eae1bc2b36d111ec43a2e5b6f07755`
[fission-core-1.6.0.yaml](https://github.com/fission/fission/releases/download/1.6.0/fission-core-1.6.0.yaml) | `7640294c0ab7a3192f147c1f17a3949c2f2c76ec9be70509f7a13ad7083318b6`
[fission-all-1.6.0-minikube.yaml](https://github.com/fission/fission/releases/download/1.6.0/fission-all-1.6.0-minikube.yaml) | `eb07b2ead721b98d4f26ffefcd03497125e8d017d82f1eaf9c0c2541d7c80288`
[fission-core-1.6.0.tgz](https://github.com/fission/fission/releases/download/1.6.0/fission-core-1.6.0.tgz) | `1dfbbc8dfa2f28bb498d3729a4bb608772d32b9644011f5e87d0dd79baa07c23`
[fission-all-1.6.0.tgz](https://github.com/fission/fission/releases/download/1.6.0/fission-all-1.6.0.tgz) | `83094e61b10a8bbc6ef7697b0f00f9f52f3c2cfb1590fcb7890eb1327d1b3c94`
[fission-cli-osx](https://github.com/fission/fission/releases/download/1.6.0/fission-cli-osx) | `0f15869c30988667e04780e01ad2ab3c5a51578c6dbbe7a54be43496d1973be5`
[fission-cli-windows.exe](https://github.com/fission/fission/releases/download/1.6.0/fission-cli-windows.exe) | `56f09af9cb26c8f4cf9cd27d06865f2f4b4138ea19fe13e9a0e8d17a2a2d405e`
[fission-cli-linux](https://github.com/fission/fission/releases/download/1.6.0/fission-cli-linux) | `90b1c14b584d52452089af351d5f615730fb5d5dd626884ec9b2808904b12df8`
[fission-all-1.7.0-rc.1-minikube.yaml](https://github.com/fission/fission/releases/download/1.7.0-rc.1/fission-all-1.7.0-rc.1-minikube.yaml) | `360645acc1a13dfce5e739ac9cf1abd3cc0aebed612a3a010eb4db668ba9a34b`
[fission-core-1.7.0-rc.1-minikube.yaml](https://github.com/fission/fission/releases/download/1.7.0-rc.1/fission-core-1.7.0-rc.1-minikube.yaml) | `8564fe7d7acdcb17baad49f4f3c20b23bb101bd6e550a84a408eeeecceb7d00e`
[fission-core-1.7.0-rc.1.yaml](https://github.com/fission/fission/releases/download/1.7.0-rc.1/fission-core-1.7.0-rc.1.yaml) | `1e49e381b08e71454ae330eca958842957d9401a6d26bdfb028ec09338ad68fe`
[fission-all-1.7.0-rc.1.yaml](https://github.com/fission/fission/releases/download/1.7.0-rc.1/fission-all-1.7.0-rc.1.yaml) | `6d12dc90d0a4395eed2397362789843e894534ee446a08f411aa63de92f400fb`
[fission-all-1.7.0-rc.1.tgz](https://github.com/fission/fission/releases/download/1.7.0-rc.1/fission-all-1.7.0-rc.1.tgz) | `be78fd00b1a8df028a41b000838c00e7e16eea25b8a6d0014ee841b8ed73e9c7`
[fission-core-1.7.0-rc.1.tgz](https://github.com/fission/fission/releases/download/1.7.0-rc.1/fission-core-1.7.0-rc.1.tgz) | `07cb5287d0624af5ab025821a1222e764f6e9634b82003b23a46910f9de2f80a`
[fission-cli-osx](https://github.com/fission/fission/releases/download/1.7.0-rc.1/fission-cli-osx) | `d68939d54a5e9c9c4bc6bb5477e9e11591514d4c3dcf371bb4272b8f41915d00`
[fission-cli-windows.exe](https://github.com/fission/fission/releases/download/1.7.0-rc.1/fission-cli-windows.exe) | `adc30a600a150b78c8defdaae97f8d7aa4d3bc27b0dd71ff1b793e36bfc25437`
[fission-cli-linux](https://github.com/fission/fission/releases/download/1.7.0-rc.1/fission-cli-linux) | `5b02b0271568460de8ae7d3090e89cef16031fa8ac9e3e3b179445596bbbeca5`
# Change Log
## [1.6.0](https://github.com/fission/fission/tree/1.6.0) (2019-10-10)
[Full Changelog](https://github.com/fission/fission/compare/1.5.0...1.6.0)
## [v1.7.0-rc.1](https://github.com/fission/fission/tree/v1.7.0-rc.1) (2019-11-18)
[Full Changelog](https://github.com/fission/fission/compare/1.6.0...v1.7.0-rc.1)
**Merged pull requests:**
- Update release-builder go and helm version [\#1418](https://github.com/fission/fission/pull/1418) ([life1347](https://github.com/life1347))
- Fix failed to find init\_tools.sh [\#1417](https://github.com/fission/fission/pull/1417) ([life1347](https://github.com/life1347))
- Support semantic version tags [\#1416](https://github.com/fission/fission/pull/1416) ([life1347](https://github.com/life1347))
- Show warning when referencing nonexistent resources in spec [\#1415](https://github.com/fission/fission/pull/1415) ([life1347](https://github.com/life1347))
- Add resource info to output message when saving spec file [\#1414](https://github.com/fission/fission/pull/1414) ([life1347](https://github.com/life1347))
- Always embed given URL in the archive [\#1413](https://github.com/fission/fission/pull/1413) ([life1347](https://github.com/life1347))
- Fix wrong command and flag usage description [\#1412](https://github.com/fission/fission/pull/1412) ([life1347](https://github.com/life1347))
- Add --spec to package command [\#1411](https://github.com/fission/fission/pull/1411) ([life1347](https://github.com/life1347))
- Drop unreleased features \(record & replay\) [\#1406](https://github.com/fission/fission/pull/1406) ([life1347](https://github.com/life1347))
- Build error formatting on fission spec apply --wait [\#1403](https://github.com/fission/fission/pull/1403) ([life1347](https://github.com/life1347))
- Refactor controller client package [\#1402](https://github.com/fission/fission/pull/1402) ([life1347](https://github.com/life1347))
- Fix fn test failed to query logs from log database [\#1401](https://github.com/fission/fission/pull/1401) ([life1347](https://github.com/life1347))
- Skip trace for router healthz endpoint [\#1400](https://github.com/fission/fission/pull/1400) ([life1347](https://github.com/life1347))
- Set jaeger collector endpoint as an environment variable [\#1399](https://github.com/fission/fission/pull/1399) ([life1347](https://github.com/life1347))
- Replace deprecated serviceAccount with serviceAccountName [\#1398](https://github.com/fission/fission/pull/1398) ([life1347](https://github.com/life1347))
- Fix helm pre-upgrade check failure problem [\#1397](https://github.com/fission/fission/pull/1397) ([life1347](https://github.com/life1347))
- Prettify console output message [\#1396](https://github.com/fission/fission/pull/1396) ([life1347](https://github.com/life1347))
- fix typo [\#1395](https://github.com/fission/fission/pull/1395) ([jjmengze](https://github.com/jjmengze))
- Reorder command flags and add missing flags [\#1394](https://github.com/fission/fission/pull/1394) ([life1347](https://github.com/life1347))
- Fix CLI exits with status 0 when error occurs [\#1393](https://github.com/fission/fission/pull/1393) ([life1347](https://github.com/life1347))
- Poolmanager wait for function pod till specialization timeout [\#1392](https://github.com/fission/fission/pull/1392) ([life1347](https://github.com/life1347))
- Replace flag text with const [\#1391](https://github.com/fission/fission/pull/1391) ([life1347](https://github.com/life1347))
- Update prometheus version and disable it from the default installation [\#1389](https://github.com/fission/fission/pull/1389) ([life1347](https://github.com/life1347))
- Fix helm shows "Not a table" issue when install Fission [\#1387](https://github.com/fission/fission/pull/1387) ([life1347](https://github.com/life1347))
- Fix githook not aborting push if error occurs [\#1386](https://github.com/fission/fission/pull/1386) ([life1347](https://github.com/life1347))
- Migrate from urfave/cli to cobra [\#1385](https://github.com/fission/fission/pull/1385) ([life1347](https://github.com/life1347))
- Update maintainer info [\#1383](https://github.com/fission/fission/pull/1383) ([life1347](https://github.com/life1347))
- Update Makefile and add git pre-push hook [\#1382](https://github.com/fission/fission/pull/1382) ([life1347](https://github.com/life1347))
- Update staticcheck version and fix all warnings [\#1381](https://github.com/fission/fission/pull/1381) ([life1347](https://github.com/life1347))
- Make CLI functions return error instead of fatal out [\#1379](https://github.com/fission/fission/pull/1379) ([life1347](https://github.com/life1347))
- Refactor record command [\#1378](https://github.com/fission/fission/pull/1378) ([life1347](https://github.com/life1347))
- Fix reverse proxy shows 404 not found when Istio enabled [\#1377](https://github.com/fission/fission/pull/1377) ([life1347](https://github.com/life1347))
- Refactor time trigger command [\#1376](https://github.com/fission/fission/pull/1376) ([life1347](https://github.com/life1347))
- Refactor mqtrigger command [\#1375](https://github.com/fission/fission/pull/1375) ([life1347](https://github.com/life1347))
- added example for builder podspec [\#1374](https://github.com/fission/fission/pull/1374) ([viveksinghggits](https://github.com/viveksinghggits))
- Refactor function command [\#1372](https://github.com/fission/fission/pull/1372) ([life1347](https://github.com/life1347))
- Allow to set API type for tensorflow serving environment [\#1371](https://github.com/fission/fission/pull/1371) ([life1347](https://github.com/life1347))
- Refactor canary config command [\#1370](https://github.com/fission/fission/pull/1370) ([life1347](https://github.com/life1347))
- PodSpec support in environment builder [\#1369](https://github.com/fission/fission/pull/1369) ([viveksinghggits](https://github.com/viveksinghggits))
- Fix utility function uses the wrong flag text to get value [\#1368](https://github.com/fission/fission/pull/1368) ([life1347](https://github.com/life1347))
- Refactor HTTP trigger command [\#1367](https://github.com/fission/fission/pull/1367) ([life1347](https://github.com/life1347))
- Update READEME link and add back the basic usage [\#1366](https://github.com/fission/fission/pull/1366) ([life1347](https://github.com/life1347))
- Refactor kubewatch command [\#1365](https://github.com/fission/fission/pull/1365) ([life1347](https://github.com/life1347))
- Fix accidentally removed timestamp when listing package [\#1364](https://github.com/fission/fission/pull/1364) ([life1347](https://github.com/life1347))
- Move route creation to function [\#1362](https://github.com/fission/fission/pull/1362) ([life1347](https://github.com/life1347))
- Remove UID from CLI output [\#1361](https://github.com/fission/fission/pull/1361) ([life1347](https://github.com/life1347))
- Allow using URL as archive source when creating functions [\#1360](https://github.com/fission/fission/pull/1360) ([life1347](https://github.com/life1347))
- Refactor plugin & version subcommands [\#1359](https://github.com/fission/fission/pull/1359) ([life1347](https://github.com/life1347))
- Provide secrets and configmaps while updating the functions [\#1358](https://github.com/fission/fission/pull/1358) ([viveksinghggits](https://github.com/viveksinghggits))
- Update fission architecture doc [\#1356](https://github.com/fission/fission/pull/1356) ([life1347](https://github.com/life1347))
- calling the function that handles kafka messages, asynchronously [\#1355](https://github.com/fission/fission/pull/1355) ([viveksinghggits](https://github.com/viveksinghggits))
- Fix release script tags wrong image name & sed problem on Linux [\#1353](https://github.com/fission/fission/pull/1353) ([life1347](https://github.com/life1347))
- Update CHANGELOG for 1.6.0 [\#1352](https://github.com/fission/fission/pull/1352) ([life1347](https://github.com/life1347))
- Replace `AlwaysSample` with `ProbabilitySampler` in router \(\#1215\) [\#1348](https://github.com/fission/fission/pull/1348) ([ccamel](https://github.com/ccamel))
- Refactor package CLI command [\#1345](https://github.com/fission/fission/pull/1345) ([life1347](https://github.com/life1347))
## [1.6.0](https://github.com/fission/fission/tree/1.6.0) (2019-10-10)
[Full Changelog](https://github.com/fission/fission/compare/v1.6.0...1.6.0)
## [v1.6.0](https://github.com/fission/fission/tree/v1.6.0) (2019-10-10)
[Full Changelog](https://github.com/fission/fission/compare/1.5.0...v1.6.0)
**Merged pull requests:**
- Fission 1.6.0 [\#1351](https://github.com/fission/fission/pull/1351) ([life1347](https://github.com/life1347))
- Move statefulset to apps/v1 [\#1350](https://github.com/fission/fission/pull/1350) ([life1347](https://github.com/life1347))
- Fix newdeploy failed to find serviceEntry in cache [\#1349](https://github.com/fission/fission/pull/1349) ([life1347](https://github.com/life1347))
- Support encoded path in router [\#1347](https://github.com/fission/fission/pull/1347) ([life1347](https://github.com/life1347))
- Support custom private go vendor [\#1346](https://github.com/fission/fission/pull/1346) ([ti](https://github.com/ti))
- Support custom private go vendor [\#1346](https://github.com/fission/fission/pull/1346) ([life1347](https://github.com/life1347))
- Fix the namespace mismatch problem when deploying with a single YAML file [\#1344](https://github.com/fission/fission/pull/1344) ([life1347](https://github.com/life1347))
- Move charts executor service to the right place [\#1343](https://github.com/fission/fission/pull/1343) ([life1347](https://github.com/life1347))
- Allow to deploy router as DaemonSet [\#1342](https://github.com/fission/fission/pull/1342) ([life1347](https://github.com/life1347))
- add package filter feature [\#1341](https://github.com/fission/fission/pull/1341) ([MengZn](https://github.com/MengZn))
- add package filter feature [\#1341](https://github.com/fission/fission/pull/1341) ([jjmengze](https://github.com/jjmengze))
- Bump jvm builder JDK version [\#1340](https://github.com/fission/fission/pull/1340) ([life1347](https://github.com/life1347))
- Fix executor doesn't apply user-configured container spec correctly [\#1339](https://github.com/fission/fission/pull/1339) ([life1347](https://github.com/life1347))
- Allow to add annotations to router service in helm chart [\#1338](https://github.com/fission/fission/pull/1338) ([prabhu43](https://github.com/prabhu43))
@@ -90,7 +155,6 @@ filename | sha256 hash
- Update tensorflow serving image name in env README [\#1256](https://github.com/fission/fission/pull/1256) ([life1347](https://github.com/life1347))
- V1.4.1 [\#1255](https://github.com/fission/fission/pull/1255) ([vishal-biyani](https://github.com/vishal-biyani))
- Fix environment version validation [\#1253](https://github.com/fission/fission/pull/1253) ([davidsmf](https://github.com/davidsmf))
- Allow to set deployment config uid during initialization [\#1249](https://github.com/fission/fission/pull/1249) ([life1347](https://github.com/life1347))
- Add swagger \(OpenAPI 2.0\) support [\#1245](https://github.com/fission/fission/pull/1245) ([life1347](https://github.com/life1347))
- Update go dependencies [\#1240](https://github.com/fission/fission/pull/1240) ([life1347](https://github.com/life1347))
@@ -101,6 +165,7 @@ filename | sha256 hash
- Fix wrongly replace spec api version [\#1254](https://github.com/fission/fission/pull/1254) ([life1347](https://github.com/life1347))
- Revert change of product name in README [\#1250](https://github.com/fission/fission/pull/1250) ([davidsmf](https://github.com/davidsmf))
- Allow to set deployment config uid during initialization [\#1249](https://github.com/fission/fission/pull/1249) ([life1347](https://github.com/life1347))
- Analytics env fix in chart [\#1247](https://github.com/fission/fission/pull/1247) ([vishal-biyani](https://github.com/vishal-biyani))
- Fix CI unable to start test due to the same travis build ID [\#1246](https://github.com/fission/fission/pull/1246) ([life1347](https://github.com/life1347))
- V1.4.0 [\#1244](https://github.com/fission/fission/pull/1244) ([vishal-biyani](https://github.com/vishal-biyani))
@@ -146,9 +211,7 @@ filename | sha256 hash
- Add readiness/liveness probes to nat-streaming [\#1199](https://github.com/fission/fission/pull/1199) ([life1347](https://github.com/life1347))
- Update bug issue templates [\#1198](https://github.com/fission/fission/pull/1198) ([life1347](https://github.com/life1347))
- Add static code analysis to CI test [\#1197](https://github.com/fission/fission/pull/1197) ([life1347](https://github.com/life1347))
- Analytics bugfix [\#1195](https://github.com/fission/fission/pull/1195) ([soamvasani](https://github.com/soamvasani))
- Add Terraform configuration and upgrade helm version [\#1194](https://github.com/fission/fission/pull/1194) ([darkgerm](https://github.com/darkgerm))
- Show warning message if spec alters poolsize while env version \< 3 [\#1193](https://github.com/fission/fission/pull/1193) ([life1347](https://github.com/life1347))
- Move packages to proejct/pkg to follow go project folder structure convention [\#1190](https://github.com/fission/fission/pull/1190) ([life1347](https://github.com/life1347))
- router analytics -- close http response body [\#1180](https://github.com/fission/fission/pull/1180) ([soamvasani](https://github.com/soamvasani))
- Remove prometheus server connectivity test during controller initialization [\#1179](https://github.com/fission/fission/pull/1179) ([life1347](https://github.com/life1347))
@@ -158,7 +221,6 @@ filename | sha256 hash
- Refactor test framework [\#1128](https://github.com/fission/fission/pull/1128) ([darkgerm](https://github.com/darkgerm))
- Pod specs [\#1106](https://github.com/fission/fission/pull/1106) ([vishal-biyani](https://github.com/vishal-biyani))
- Allow non-toplevel modules in python environment [\#1042](https://github.com/fission/fission/pull/1042) ([soamvasani](https://github.com/soamvasani))
- Created dotnet2.0 Builder Image and Added /v2/specialized Endpoint to dotnet2.0 Envrionment [\#1001](https://github.com/fission/fission/pull/1001) ([paraspatidar](https://github.com/paraspatidar))
## [1.2.1](https://github.com/fission/fission/tree/1.2.1) (2019-05-09)
[Full Changelog](https://github.com/fission/fission/compare/1.2.0...1.2.1)
@@ -174,6 +236,8 @@ filename | sha256 hash
**Merged pull requests:**
- Analytics bugfix [\#1195](https://github.com/fission/fission/pull/1195) ([soamvasani](https://github.com/soamvasani))
- Show warning message if spec alters poolsize while env version \< 3 [\#1193](https://github.com/fission/fission/pull/1193) ([life1347](https://github.com/life1347))
- DRY up fetcher configuration [\#1168](https://github.com/fission/fission/pull/1168) ([vishal-biyani](https://github.com/vishal-biyani))
- Add simple anonymous usage metrics [\#1167](https://github.com/fission/fission/pull/1167) ([soamvasani](https://github.com/soamvasani))
- Fix the logger not working [\#1166](https://github.com/fission/fission/pull/1166) ([darkgerm](https://github.com/darkgerm))
@@ -188,6 +252,7 @@ filename | sha256 hash
- support KUBECONFIG with multiple kube config files [\#1126](https://github.com/fission/fission/pull/1126) ([grounded042](https://github.com/grounded042))
- Function update after change in env [\#1116](https://github.com/fission/fission/pull/1116) ([vishal-biyani](https://github.com/vishal-biyani))
- Add configurable timeout to fission function test [\#1091](https://github.com/fission/fission/pull/1091) ([erwinvaneyk](https://github.com/erwinvaneyk))
- Add links to examples for each Fission environment [\#1090](https://github.com/fission/fission/pull/1090) ([erwinvaneyk](https://github.com/erwinvaneyk))
## [1.1.0](https://github.com/fission/fission/tree/1.1.0) (2019-03-25)
[Full Changelog](https://github.com/fission/fission/compare/1.0.0...1.1.0)
@@ -205,11 +270,11 @@ filename | sha256 hash
- Added support for Ruby v2 Specialization [\#1101](https://github.com/fission/fission/pull/1101) ([brendanstennett](https://github.com/brendanstennett))
- V1.0.0 [\#1100](https://github.com/fission/fission/pull/1100) ([vishal-biyani](https://github.com/vishal-biyani))
- Adding annotations for prometheus scraping to fission-core [\#1098](https://github.com/fission/fission/pull/1098) ([vishal-biyani](https://github.com/vishal-biyani))
- Add links to examples for each Fission environment [\#1090](https://github.com/fission/fission/pull/1090) ([erwinvaneyk](https://github.com/erwinvaneyk))
- Switch from fluentd to fluentbit for log forwarding [\#1086](https://github.com/fission/fission/pull/1086) ([soamvasani](https://github.com/soamvasani))
- Added draft proposal for CI/CD [\#1084](https://github.com/fission/fission/pull/1084) ([vishal-biyani](https://github.com/vishal-biyani))
- \[Kafka MQT\] Add warning about Kafka version [\#1083](https://github.com/fission/fission/pull/1083) ([bhavin192](https://github.com/bhavin192))
- Bump base image version of Go environment to 1.11.4 [\#1026](https://github.com/fission/fission/pull/1026) ([life1347](https://github.com/life1347))
- Created dotnet2.0 Builder Image and Added /v2/specialized Endpoint to dotnet2.0 Envrionment [\#1001](https://github.com/fission/fission/pull/1001) ([paraspatidar](https://github.com/paraspatidar))
## [1.0.0](https://github.com/fission/fission/tree/1.0.0) (2019-02-13)
[Full Changelog](https://github.com/fission/fission/compare/1.0...1.0.0)
@@ -326,7 +391,6 @@ filename | sha256 hash
**Merged pull requests:**
- Print status with the get option. [\#907](https://github.com/fission/fission/pull/907) ([smruthi2187](https://github.com/smruthi2187))
- Fixed the spec validation UX issue [\#898](https://github.com/fission/fission/pull/898) ([vishal-biyani](https://github.com/vishal-biyani))
- Check CRD creation error instead of doing return directly [\#897](https://github.com/fission/fission/pull/897) ([life1347](https://github.com/life1347))
- Fix failed to find release-builder dockerfile & push specific tag [\#870](https://github.com/fission/fission/pull/870) ([life1347](https://github.com/life1347))
- V0.10.0 [\#868](https://github.com/fission/fission/pull/868) ([life1347](https://github.com/life1347))
@@ -347,6 +411,7 @@ filename | sha256 hash
**Merged pull requests:**
- Fixed the spec validation UX issue [\#898](https://github.com/fission/fission/pull/898) ([vishal-biyani](https://github.com/vishal-biyani))
- Fix CLI failed to setup port-forward caused by \#712 [\#867](https://github.com/fission/fission/pull/867) ([life1347](https://github.com/life1347))
- Replay recorded requests by ReqUID [\#864](https://github.com/fission/fission/pull/864) ([Amusement](https://github.com/Amusement))
- Add cleanup function to test scripts [\#863](https://github.com/fission/fission/pull/863) ([life1347](https://github.com/life1347))
@@ -470,7 +535,6 @@ filename | sha256 hash
- Changes needed for release 0.7.1 [\#622](https://github.com/fission/fission/pull/622) ([smruthi2187](https://github.com/smruthi2187))
- Add default value to cli flag [\#619](https://github.com/fission/fission/pull/619) ([life1347](https://github.com/life1347))
- Remove port forward in tests for router, controller and nats pods [\#611](https://github.com/fission/fission/pull/611) ([smruthi2187](https://github.com/smruthi2187))
- updates to changelog. [\#598](https://github.com/fission/fission/pull/598) ([smruthi2187](https://github.com/smruthi2187))
- meaningful error message when fetch request is received for a package when build is not successful. [\#661](https://github.com/fission/fission/pull/661) ([smruthi2187](https://github.com/smruthi2187))
- Delete deployment with proper delete propagation policy [\#630](https://github.com/fission/fission/pull/630) ([life1347](https://github.com/life1347))
- Fix buildmgr SEGFAULT when it failed to update package [\#626](https://github.com/fission/fission/pull/626) ([life1347](https://github.com/life1347))
@@ -485,6 +549,7 @@ filename | sha256 hash
- Prevent releasing idle connections because transport is shared. [\#609](https://github.com/fission/fission/pull/609) ([smruthi2187](https://github.com/smruthi2187))
- Fix components crash before crds creation [\#602](https://github.com/fission/fission/pull/602) ([life1347](https://github.com/life1347))
- updates to changelog. [\#598](https://github.com/fission/fission/pull/598) ([smruthi2187](https://github.com/smruthi2187))
- changes needed for release 0.7.0 [\#597](https://github.com/fission/fission/pull/597) ([smruthi2187](https://github.com/smruthi2187))
- `fission X create --spec` flags for env and trigger create commands [\#607](https://github.com/fission/fission/pull/607) ([soamvasani](https://github.com/soamvasani))
- Updating releasing guideliness with a few more details. [\#599](https://github.com/fission/fission/pull/599) ([smruthi2187](https://github.com/smruthi2187))
@@ -864,6 +929,7 @@ filename | sha256 hash
- Readme updates [\#29](https://github.com/fission/fission/pull/29) ([soamvasani](https://github.com/soamvasani))
- Add "fission function edit \<function\>" command [\#28](https://github.com/fission/fission/pull/28) ([soamvasani](https://github.com/soamvasani))
- Move client-go dependency to 1.5 [\#27](https://github.com/fission/fission/pull/27) ([soamvasani](https://github.com/soamvasani))
- fission-bundle: executable package for router, controller, poolmgr [\#17](https://github.com/fission/fission/pull/17) ([soamvasani](https://github.com/soamvasani))
## [kubecon](https://github.com/fission/fission/tree/kubecon) (2016-11-11)
**Merged pull requests:**
@@ -871,7 +937,6 @@ filename | sha256 hash
- Reap idle pods [\#20](https://github.com/fission/fission/pull/20) ([soamvasani](https://github.com/soamvasani))
- Fission CLI [\#19](https://github.com/fission/fission/pull/19) ([soamvasani](https://github.com/soamvasani))
- Fix resource store errors on empty db [\#18](https://github.com/fission/fission/pull/18) ([soamvasani](https://github.com/soamvasani))
- fission-bundle: executable package for router, controller, poolmgr [\#17](https://github.com/fission/fission/pull/17) ([soamvasani](https://github.com/soamvasani))
- Router integration with poolmgr and controller [\#16](https://github.com/fission/fission/pull/16) ([soamvasani](https://github.com/soamvasani))
- Poolmgr -- manage generic containers and their specialization [\#15](https://github.com/fission/fission/pull/15) ([soamvasani](https://github.com/soamvasani))
- Fetcher is a helper for function run containers [\#14](https://github.com/fission/fission/pull/14) ([soamvasani](https://github.com/soamvasani))
+2 -2
View File
@@ -43,10 +43,10 @@ Parameter | Description | Default
`routerServiceType` | Type of Fission Router service to use. For minikube, set this to NodePort, elsewhere use LoadBalancer or ClusterIP. | `LoadBalancer`
`repository` | Image base repository | `index.docker.io`
`image` | Fission image repository | `fission/fission-bundle`
`imageTag` | Fission image tag | `1.7.0-rc.1`
`imageTag` | Fission image tag | `1.7.0-rc.2`
`pullPolicy` | Image pull policy | `IfNotPresent`
`fetcherImage` | Fission fetcher repository | `fission/fetcher`
`fetcherImageTag` | Fission fetcher image tag | `1.7.0-rc.1`
`fetcherImageTag` | Fission fetcher image tag | `1.7.0-rc.2`
`controllerPort` | Fission Controller service port | `31313`
`routerPort` | Fission Router service port | ` 31314`
`functionNamespace` | Namespace in which to run fission functions (this is different from the release namespace) | `fission-function`
+2 -2
View File
@@ -1,6 +1,6 @@
apiVersion: v1
name: fission-all
version: 1.7.0-rc.1
version: 1.7.0-rc.2
description: Fission is a fast serverless framework for Kubernetes.
keywords:
- fission
@@ -12,4 +12,4 @@ maintainers:
- name: Ta Ching Chen
email: hello@tachingchen.com
engine: gotpl
appVersion: 1.7.0-rc.1
appVersion: 1.7.0-rc.2
+2 -2
View File
@@ -2,5 +2,5 @@ dependencies:
- name: prometheus
repository: https://kubernetes-charts.storage.googleapis.com
version: 9.3.1
digest: sha256:33395db6ac17f57dab0e60a47a3e029c59aa4a58fb1695c73b8ea6f0754eaf0c
generated: 2019-11-09T02:56:08.384381+08:00
digest: sha256:89a63a5f01c08032106a94c59fc7a5929e449458bf5fd06d86c8b048ac628bf4
generated: "2019-11-18T10:16:20.334338911Z"
@@ -534,10 +534,8 @@ spec:
]
ports:
- containerPort: 4222
hostPort: 4222
protocol: TCP
- containerPort: 4223
hostPort: 4223
protocol: TCP
readinessProbe:
httpGet:
@@ -45,6 +45,10 @@ spec:
- name: container-log
mountPath: /var/log/
readOnly: false
{{- if .Values.logger.enableSecurityContext }}
securityContext:
privileged: true
{{- end }}
containers:
- name: logger
{{ if .Values.repository }}
@@ -68,6 +72,10 @@ spec:
- name: docker-log
mountPath: /var/lib/docker/containers
readOnly: true
{{- if .Values.logger.enableSecurityContext }}
securityContext:
privileged: true
{{- end }}
- name: fluentbit
{{- if .Values.repository }}
image: "{{ .Values.logger.fluentdImageRepository }}/{{ .Values.logger.fluentdImage }}:{{ .Values.logger.fluentdImageTag }}"
@@ -96,6 +104,10 @@ spec:
key: password
- name: LOG_PATH
value: /var/log/fission/*.log
{{- if .Values.logger.enableSecurityContext }}
securityContext:
privileged: true
{{- end }}
volumeMounts:
- name: container-log
mountPath: /var/log/
+9 -2
View File
@@ -20,13 +20,13 @@ image: fission/fission-bundle
pullPolicy: IfNotPresent
## Fission image version
imageTag: 1.7.0-rc.1
imageTag: 1.7.0-rc.2
## Fission fetcher repository
fetcherImage: fission/fetcher
## Fission fetcher image version
fetcherImageTag: 1.7.0-rc.1
fetcherImageTag: 1.7.0-rc.2
## Port at which Fission controller service should be exposed
controllerPort: 31313
@@ -57,6 +57,13 @@ logger:
fluentdImageRepository: index.docker.io
fluentdImage: fluent/fluent-bit
fluentdImageTag: 1.0.4
## Fluent-bit writes/reads its own sqlite database to record a history of tracked
## files and a state of offsets, this is very useful to resume a state if the ser-
## vice is restarted. For Kubernetes environment with constraints like OpenShift,
## the containers are limited to write hostPath volume. Hence, we have to enable
## security context and set privileged to true.
enableSecurityContext: false
## Router config
router:
+2 -2
View File
@@ -1,6 +1,6 @@
apiVersion: v1
name: fission-core
version: 1.7.0-rc.1
version: 1.7.0-rc.2
description: Fission is a fast serverless framework for Kubernetes.
keywords:
- fission
@@ -12,4 +12,4 @@ maintainers:
- name: Ta Ching Chen
email: hello@tachingchen.com
engine: gotpl
appVersion: 1.7.0-rc.1
appVersion: 1.7.0-rc.2
+2 -2
View File
@@ -2,5 +2,5 @@ dependencies:
- name: prometheus
repository: https://kubernetes-charts.storage.googleapis.com
version: 9.3.1
digest: sha256:33395db6ac17f57dab0e60a47a3e029c59aa4a58fb1695c73b8ea6f0754eaf0c
generated: 2019-11-09T02:56:58.282692+08:00
digest: sha256:89a63a5f01c08032106a94c59fc7a5929e449458bf5fd06d86c8b048ac628bf4
generated: "2019-11-18T10:16:25.945208326Z"
+2 -2
View File
@@ -17,7 +17,7 @@ repository: index.docker.io
image: fission/fission-bundle
## Fission image version
imageTag: 1.7.0-rc.1
imageTag: 1.7.0-rc.2
## Image pull policy
pullPolicy: IfNotPresent
@@ -26,7 +26,7 @@ pullPolicy: IfNotPresent
fetcherImage: fission/fetcher
## Fission fetcher image version
fetcherImageTag: 1.7.0-rc.1
fetcherImageTag: 1.7.0-rc.2
## Port at which Fission controller service should be exposed
controllerPort: 31313
+5 -3
View File
@@ -37,7 +37,7 @@ func App() *cobra.Command {
Use: "fission",
Long: usage,
//SilenceUsage: true,
PreRunE: wrapper.Wrapper(
PersistentPreRunE: wrapper.Wrapper(
func(input cli.Input) error {
console.Verbosity = input.Int(flagkey.Verbosity)
return nil
@@ -53,7 +53,7 @@ func App() *cobra.Command {
})
wrapper.SetFlags(rootCmd, flag.FlagSet{
Optional: []flag.Flag{flag.GlobalServer, flag.GlobalVerbosity},
Global: []flag.Flag{flag.GlobalServer, flag.GlobalVerbosity},
})
groups := helptemplate.CommandGroups{}
@@ -64,7 +64,9 @@ func App() *cobra.Command {
groups = append(groups, helptemplate.CreateCmdGroup("Other Commands", support.Commands(), version.Commands()))
groups.Add(rootCmd)
helptemplate.ActsAsRootCommand(rootCmd, nil, groups...)
flagExposer := helptemplate.ActsAsRootCommand(rootCmd, nil, groups...)
// show global options in usage
flagExposer.ExposeFlags(rootCmd, flagkey.Server, flagkey.Verbosity)
return rootCmd
}
+2 -3
View File
@@ -75,8 +75,7 @@ a. Switch to fission/fission-charts repo and run `index.sh` in that repo. (Don'
b. You should have index.yaml and fission-core-*, fission-all-* charts as change, add, commit and push it to repo.
8. Go to Github Web UI --> Releases, here you will notice a pre-release for the version x.y.z you just created.
9. Go to Github Web UI --> Releases, here you will notice a pre-release for the version x.y.z you just created.
Edit it and add links to:
@@ -88,7 +87,7 @@ Before you save the release - UNCHECK the "This is a pre-release" checkbox. This
Updating Fission Docs (https://github.com/fission/docs.fission.io)
------------------------------
9. Documentation Update
10. Documentation Update
a. Merge documentation PRs that are peer reviewed and get latest master locally.
+4 -2
View File
@@ -198,8 +198,8 @@ build_charts() {
do
# https://github.com/kubernetes/helm/issues/1732
helm init --client-only
helm package -u $c/
mv *.tgz $BUILDDIR/charts/
helm package -u $c/
mv *.tgz $BUILDDIR/charts/
done
popd
}
@@ -226,6 +226,8 @@ build_yamls() {
helm template ${c} -n ${releaseName} --namespace fission --set analytics=false,analyticsNonHelmInstall=true,serviceType=NodePort,routerServiceType=NodePort > ${c}-${version}-minikube.yaml
# for environments that support LoadBalancer
helm template ${c} -n ${releaseName} --namespace fission --set analytics=false,analyticsNonHelmInstall=true > ${c}-${version}.yaml
# for OpenShift
helm template ${c} -n ${releaseName} --namespace fission --set analytics=false,analyticsNonHelmInstall=true,logger.enableSecurityContext=true,prometheus.enabled=false > ${c}-${version}-openshift.yaml
# copy yaml files to build directory
mv *.yaml ${BUILDDIR}/yamls/
+18 -9
View File
@@ -14,8 +14,8 @@ check_branch() {
curr_branch=$(git rev-parse --abbrev-ref HEAD)
if [ $curr_branch != "release-${version}" ]
then
echo "Not on release-${version} branch."
exit 1
echo "Not on release-${version} branch."
exit 1
fi
}
@@ -23,8 +23,8 @@ check_branch() {
check_clean() {
if ! git diff-index --quiet HEAD --
then
echo "Unclean tree"
exit 1
echo "Unclean tree"
exit 1
fi
}
@@ -150,13 +150,22 @@ push_all() {
tag_and_release() {
local version=$1
local gittag="v${version}"
local gittag=$version
local prefix="v"
local gopkgtag=${version/#/${prefix}}
if [[ ${version} == v* ]]; # if version starts with "v", don't append prefix.
then
gopkgtag=${version}
fi
# tag the release
git tag $gittag
git tag -a $gopkgtag -m "Fission $gopkgtag"
# push tag
git push origin $gittag
git push origin $gopkgtag
# create gh release
gothub release \
@@ -170,7 +179,7 @@ tag_and_release() {
attach_github_release_cli() {
local version=$1
local gittag="v${version}"
local gittag=$version
# cli
echo "Uploading osx cli"
gothub upload \
@@ -202,7 +211,7 @@ attach_github_release_cli() {
attach_github_release_charts() {
local version=$1
local gittag="v${version}"
local gittag=$version
# helm charts
gothub upload \
@@ -225,7 +234,7 @@ attach_github_release_charts() {
attach_github_release_yamls() {
local version=$1
local gittag="v${version}"
local gittag=$version
for c in fission-all fission-core
do
+2 -2
View File
@@ -54,8 +54,8 @@ const (
)
const (
ExecutorTypePoolmgr = "poolmgr"
ExecutorTypeNewdeploy = "newdeploy"
ExecutorTypePoolmgr ExecutorType = "poolmgr"
ExecutorTypeNewdeploy ExecutorType = "newdeploy"
)
const (
+4 -1
View File
@@ -381,7 +381,6 @@ type (
TargetCPUPercent int
// This is the timeout setting for executor to wait for pod specialization.
// Currently, only newdeploy utilizes this value.
SpecializationTimeout int
}
@@ -529,6 +528,10 @@ type (
// or unarchived file should be placed, which is then used by specialize handler.
// (This is mainly for the JVM environment because .jar is one kind of zip archive.)
KeepArchive bool `json:"keeparchive"`
// ImagePullSecret is the secret for Kubernetes to pull an image from a
// private registry.
ImagePullSecret string `json:"imagepullsecret"`
}
AllowedFunctionsPerContainer string
+1 -2
View File
@@ -495,7 +495,6 @@ func (envw *environmentWatcher) createBuilderDeployment(env *fv1.Environment, ns
}
finalPodSpec, err := util.MergePodSpec(&podSpec, env.Spec.Builder.PodSpec)
if err != nil {
return nil, err
}
@@ -516,7 +515,7 @@ func (envw *environmentWatcher) createBuilderDeployment(env *fv1.Environment, ns
Labels: sel,
Annotations: podAnnotations,
},
Spec: *finalPodSpec,
Spec: *(util.ApplyImagePullSecret(env.Spec.ImagePullSecret, *finalPodSpec)),
},
},
}
+53 -20
View File
@@ -17,7 +17,6 @@ limitations under the License.
package executor
import (
"context"
"encoding/json"
"fmt"
"io/ioutil"
@@ -25,6 +24,8 @@ import (
"strings"
"github.com/gorilla/mux"
"github.com/hashicorp/go-multierror"
"github.com/pkg/errors"
"go.opencensus.io/plugin/ochttp"
"go.uber.org/zap"
k8serrors "k8s.io/apimachinery/pkg/api/errors"
@@ -32,6 +33,7 @@ import (
fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
ferror "github.com/fission/fission/pkg/error"
"github.com/fission/fission/pkg/executor/client"
)
func (executor *Executor) getServiceForFunctionApi(w http.ResponseWriter, r *http.Request) {
@@ -88,9 +90,15 @@ func (executor *Executor) getServiceForFunction(fn *fv1.Function) (string, error
zap.String("function_name", fn.Metadata.Name),
zap.String("function_namespace", fn.Metadata.Namespace))
fsvc, err := executor.fsCache.GetByFunction(&fn.Metadata)
t := fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType
et, exists := executor.executorTypes[t]
if !exists {
return "", errors.Errorf("Unknown executor type '%v'", t)
}
fsvc, err := et.GetFuncSvcFromCache(fn)
if err == nil {
if executor.isValidAddress(fsvc) {
if et.IsValid(fsvc) {
// Cached, return svc address
return fsvc.Address, nil
} else {
@@ -98,7 +106,7 @@ func (executor *Executor) getServiceForFunction(fn *fv1.Function) (string, error
zap.String("function_name", fn.Metadata.Name),
zap.String("function_namespace", fn.Metadata.Namespace),
zap.String("address", fsvc.Address))
executor.fsCache.DeleteEntry(fsvc)
et.DeleteFuncSvcFromCache(fsvc)
}
}
@@ -115,25 +123,57 @@ func (executor *Executor) getServiceForFunction(fn *fv1.Function) (string, error
}
// find funcSvc and update its atime
// TODO: Deprecated tapService
func (executor *Executor) tapService(w http.ResponseWriter, r *http.Request) {
// only for upgrade compatibility
w.WriteHeader(http.StatusOK)
}
// find funcSvc and update its atime
func (executor *Executor) tapServices(w http.ResponseWriter, r *http.Request) {
body, err := ioutil.ReadAll(r.Body)
if err != nil {
executor.logger.Error("failed to read tap service request", zap.Error(err))
http.Error(w, "Failed to read request", http.StatusInternalServerError)
return
}
svcName := string(body)
svcHost := strings.TrimPrefix(svcName, "http://")
err = executor.fsCache.TouchByAddress(svcHost)
tapSvcReqs := []client.TapServiceRequest{}
err = json.Unmarshal(body, &tapSvcReqs)
if err != nil {
executor.logger.Error("error tapping function service",
executor.logger.Error("failed to decode tap service request",
zap.Error(err),
zap.String("service", svcName),
zap.String("host", svcHost))
zap.String("request-payload", string(body)))
http.Error(w, "Failed to decode tap service request", http.StatusBadRequest)
return
}
errs := &multierror.Error{}
for _, req := range tapSvcReqs {
svcHost := strings.TrimPrefix(req.ServiceUrl, "http://")
et, exists := executor.executorTypes[req.FnExecutorType]
if !exists {
errs = multierror.Append(errs,
errors.Errorf("error tapping service due to unknown executor type '%v' found",
req.FnExecutorType))
continue
}
err = et.TapService(svcHost)
if err != nil {
errs = multierror.Append(errs,
errors.Wrapf(err, "'%v' failed to tap function '%v' in '%v' with service url '%v'",
req.FnMetadata.Name, req.FnMetadata.Namespace, req.ServiceUrl, req.FnExecutorType))
}
}
if errs.ErrorOrNil() != nil {
executor.logger.Error("error tapping function service", zap.Error(errs))
http.Error(w, "Not found", http.StatusNotFound)
return
}
w.WriteHeader(http.StatusOK)
}
@@ -144,21 +184,14 @@ func (executor *Executor) healthHandler(w http.ResponseWriter, r *http.Request)
func (executor *Executor) GetHandler() http.Handler {
r := mux.NewRouter()
r.HandleFunc("/v2/getServiceForFunction", executor.getServiceForFunctionApi).Methods("POST")
r.HandleFunc("/v2/tapService", executor.tapService).Methods("POST")
r.HandleFunc("/v2/tapService", executor.tapService).Methods("POST") // for backward compatibility
r.HandleFunc("/v2/tapServices", executor.tapServices).Methods("POST")
r.HandleFunc("/healthz", executor.healthHandler).Methods("GET")
return r
}
func (executor *Executor) Serve(port int) {
executor.logger.Info("starting executor", zap.Int("port", port))
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
executor.ndm.Run(ctx)
executor.gpm.Run(ctx)
executor.cms.Run(ctx)
executor.logger.Info("starting executor API", zap.Int("port", port))
address := fmt.Sprintf(":%v", port)
err := http.ListenAndServe(address, &ochttp.Handler{
Handler: executor.GetHandler(),
+60 -28
View File
@@ -32,23 +32,31 @@ import (
"golang.org/x/net/context/ctxhttp"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
ferror "github.com/fission/fission/pkg/error"
)
type Client struct {
logger *zap.Logger
executorUrl string
tappedByUrl map[string]bool
requestChan chan string
httpClient *http.Client
}
type (
Client struct {
logger *zap.Logger
executorUrl string
tappedByUrl map[string]TapServiceRequest
requestChan chan TapServiceRequest
httpClient *http.Client
}
TapServiceRequest struct {
FnMetadata metav1.ObjectMeta
FnExecutorType fv1.ExecutorType
ServiceUrl string
}
)
func MakeClient(logger *zap.Logger, executorUrl string) *Client {
c := &Client{
logger: logger.Named("executor_client"),
executorUrl: strings.TrimSuffix(executorUrl, "/"),
tappedByUrl: make(map[string]bool),
requestChan: make(chan string),
tappedByUrl: make(map[string]TapServiceRequest),
requestChan: make(chan TapServiceRequest, 100),
httpClient: &http.Client{
Transport: &ochttp.Transport{},
},
@@ -87,40 +95,64 @@ func (c *Client) service() {
ticker := time.NewTicker(time.Second * 5)
for {
select {
case serviceUrl := <-c.requestChan:
c.tappedByUrl[serviceUrl] = true
case svcReq := <-c.requestChan:
c.tappedByUrl[svcReq.ServiceUrl] = svcReq
case <-ticker.C:
urls := c.tappedByUrl
c.tappedByUrl = make(map[string]bool)
if len(urls) > 0 {
go func() {
for u := range urls {
err := c._tapService(u)
if err != nil {
c.logger.Error("error tapping function service address", zap.Error(err), zap.String("address", u))
}
}
c.logger.Debug("tapped services in batch", zap.Int("service_count", len(urls)))
}()
if len(c.tappedByUrl) == 0 {
continue
}
urls := c.tappedByUrl
c.tappedByUrl = make(map[string]TapServiceRequest)
go func() {
svcReqs := []TapServiceRequest{}
for _, req := range urls {
svcReqs = append(svcReqs, req)
}
c.logger.Debug("tapped services in batch", zap.Int("service_count", len(urls)))
err := c._tapService(svcReqs)
if err != nil {
c.logger.Error("error tapping function service address", zap.Error(err))
}
}()
}
}
}
func (c *Client) TapService(serviceUrl *url.URL) {
c.requestChan <- serviceUrl.String()
func (c *Client) TapService(fnMeta metav1.ObjectMeta, executorType fv1.ExecutorType, serviceUrl *url.URL) {
c.requestChan <- TapServiceRequest{
FnMetadata: metav1.ObjectMeta{
Name: fnMeta.Name,
Namespace: fnMeta.Namespace,
ResourceVersion: fnMeta.ResourceVersion,
UID: fnMeta.UID,
},
FnExecutorType: executorType,
// service url is for executor to know which
// pod/service is currently used to serve user function.
ServiceUrl: serviceUrl.String(),
}
}
func (c *Client) _tapService(serviceUrlStr string) error {
executorUrl := c.executorUrl + "/v2/tapService"
func (c *Client) _tapService(tapSvcReqs []TapServiceRequest) error {
executorUrl := c.executorUrl + "/v2/tapServices"
resp, err := http.Post(executorUrl, "application/octet-stream", bytes.NewReader([]byte(serviceUrlStr)))
body, err := json.Marshal(tapSvcReqs)
if err != nil {
return err
}
resp, err := http.Post(executorUrl, "application/json", bytes.NewReader(body))
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
return ferror.MakeErrorFromHTTP(resp)
}
return nil
}
+17 -20
View File
@@ -20,6 +20,7 @@ import (
"context"
"time"
"github.com/pkg/errors"
"go.uber.org/zap"
apiv1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -29,8 +30,7 @@ import (
fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
"github.com/fission/fission/pkg/crd"
nd "github.com/fission/fission/pkg/executor/newdeploy"
gpm "github.com/fission/fission/pkg/executor/poolmgr"
"github.com/fission/fission/pkg/executor/executortype"
)
type (
@@ -46,10 +46,10 @@ type (
//MakeConfigSecretController makes a controller for configmaps and secrets which changes related functions
func MakeConfigSecretController(logger *zap.Logger, fissionClient *crd.FissionClient,
kubernetesClient *kubernetes.Clientset, ndm *nd.NewDeploy, gpm *gpm.GenericPoolManager) *ConfigSecretController {
kubernetesClient *kubernetes.Clientset, types map[fv1.ExecutorType]executortype.ExecutorType) *ConfigSecretController {
logger.Debug("Creating ConfigMap & Secret Controller")
_, cmcontroller := initConfigmapController(logger, fissionClient, kubernetesClient, ndm, gpm)
_, scontroller := initSecretController(logger, fissionClient, kubernetesClient, ndm, gpm)
_, cmcontroller := initConfigmapController(logger, fissionClient, kubernetesClient, types)
_, scontroller := initSecretController(logger, fissionClient, kubernetesClient, types)
cmsController := &ConfigSecretController{
logger: logger,
configmapController: cmcontroller,
@@ -66,7 +66,7 @@ func (csController *ConfigSecretController) Run(ctx context.Context) {
}
func initConfigmapController(logger *zap.Logger, fissionClient *crd.FissionClient,
kubernetesClient *kubernetes.Clientset, ndm *nd.NewDeploy, gpm *gpm.GenericPoolManager) (cache.Store, cache.Controller) {
kubernetesClient *kubernetes.Clientset, types map[fv1.ExecutorType]executortype.ExecutorType) (cache.Store, cache.Controller) {
resyncPeriod := 30 * time.Second
listWatch := cache.NewListWatchFromClient(kubernetesClient.CoreV1().RESTClient(), "configmaps", metav1.NamespaceAll, fields.Everything())
store, controller := cache.NewInformer(listWatch, &apiv1.ConfigMap{}, resyncPeriod, cache.ResourceEventHandlerFuncs{
@@ -81,12 +81,11 @@ func initConfigmapController(logger *zap.Logger, fissionClient *crd.FissionClien
zap.String("configmap_name", newCm.ObjectMeta.Name),
zap.String("configmap_namespace", newCm.ObjectMeta.Namespace))
}
funcs, err := getConfigmapRelatedFuncs(logger, &newCm.ObjectMeta, fissionClient)
if err != nil {
logger.Error("Failed to get functions related to secret", zap.String("secret_name", newCm.ObjectMeta.Name), zap.String("secret_namespace", newCm.ObjectMeta.Namespace))
logger.Error("Failed to get functions related to configmap", zap.String("configmap_name", newCm.ObjectMeta.Name), zap.String("configmap_namespace", newCm.ObjectMeta.Namespace))
}
recyclePods(logger, funcs, ndm, gpm)
recyclePods(logger, funcs, types)
}
},
})
@@ -112,7 +111,7 @@ func getConfigmapRelatedFuncs(logger *zap.Logger, m *metav1.ObjectMeta, fissionC
}
func initSecretController(logger *zap.Logger, fissionClient *crd.FissionClient,
kubernetesClient *kubernetes.Clientset, ndm *nd.NewDeploy, gpm *gpm.GenericPoolManager) (cache.Store, cache.Controller) {
kubernetesClient *kubernetes.Clientset, types map[fv1.ExecutorType]executortype.ExecutorType) (cache.Store, cache.Controller) {
resyncPeriod := 30 * time.Second
listWatch := cache.NewListWatchFromClient(kubernetesClient.CoreV1().RESTClient(), "secrets", metav1.NamespaceAll, fields.Everything())
store, controller := cache.NewInformer(listWatch, &apiv1.Secret{}, resyncPeriod, cache.ResourceEventHandlerFuncs{
@@ -128,14 +127,12 @@ func initSecretController(logger *zap.Logger, fissionClient *crd.FissionClient,
zap.String("configmap_namespace", newS.ObjectMeta.Namespace))
}
funcs, err := getSecretRelatedFuncs(logger, &newS.ObjectMeta, fissionClient)
if err != nil {
logger.Error("Failed to get functions related to secret", zap.String("secret_name", newS.ObjectMeta.Name), zap.String("secret_namespace", newS.ObjectMeta.Namespace))
}
recyclePods(logger, funcs, ndm, gpm)
recyclePods(logger, funcs, types)
}
},
})
return store, controller
@@ -160,19 +157,19 @@ func getSecretRelatedFuncs(logger *zap.Logger, m *metav1.ObjectMeta, fissionClie
return relatedFunctions, nil
}
func recyclePods(logger *zap.Logger, funcs []fv1.Function, ndm *nd.NewDeploy, gpm *gpm.GenericPoolManager) {
func recyclePods(logger *zap.Logger, funcs []fv1.Function, types map[fv1.ExecutorType]executortype.ExecutorType) {
for _, f := range funcs {
var err error
switch f.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType {
case fv1.ExecutorTypeNewdeploy:
err = ndm.RefreshFuncPods(logger, f)
case fv1.ExecutorTypePoolmgr:
err = gpm.RefreshFuncPods(logger, f)
et, exists := types[f.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType]
if exists {
err = et.RefreshFuncPods(logger, f)
} else {
err = errors.Errorf("Unknown executor type '%v'", f.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType)
}
if err != nil {
logger.Error("Failed to recycle pods for function after configmap changed",
logger.Error("Failed to recycle pods for function after configmap/secret changed",
zap.Error(err),
zap.Any("function", f))
}
+59 -46
View File
@@ -32,9 +32,10 @@ import (
fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
"github.com/fission/fission/pkg/crd"
"github.com/fission/fission/pkg/executor/cms"
"github.com/fission/fission/pkg/executor/executortype"
"github.com/fission/fission/pkg/executor/executortype/newdeploy"
"github.com/fission/fission/pkg/executor/executortype/poolmgr"
"github.com/fission/fission/pkg/executor/fscache"
"github.com/fission/fission/pkg/executor/newdeploy"
"github.com/fission/fission/pkg/executor/poolmgr"
"github.com/fission/fission/pkg/executor/reaper"
fetcherConfig "github.com/fission/fission/pkg/fetcher/config"
)
@@ -43,12 +44,10 @@ type (
Executor struct {
logger *zap.Logger
gpm *poolmgr.GenericPoolManager
ndm *newdeploy.NewDeploy
cms *cms.ConfigSecretController
executorTypes map[fv1.ExecutorType]executortype.ExecutorType
cms *cms.ConfigSecretController
fissionClient *crd.FissionClient
fsCache *fscache.FunctionServiceCache
requestChan chan *createFuncServiceRequest
fsCreateWg map[string]*sync.WaitGroup
@@ -64,21 +63,20 @@ type (
}
)
func MakeExecutor(logger *zap.Logger, gpm *poolmgr.GenericPoolManager, ndm *newdeploy.NewDeploy, cms *cms.ConfigSecretController, fissionClient *crd.FissionClient, fsCache *fscache.FunctionServiceCache) *Executor {
func MakeExecutor(logger *zap.Logger, cms *cms.ConfigSecretController,
fissionClient *crd.FissionClient, types map[fv1.ExecutorType]executortype.ExecutorType) (*Executor, error) {
executor := &Executor{
logger: logger.Named("executor"),
gpm: gpm,
ndm: ndm,
cms: cms,
fissionClient: fissionClient,
fsCache: fsCache,
executorTypes: types,
requestChan: make(chan *createFuncServiceRequest),
fsCreateWg: make(map[string]*sync.WaitGroup),
}
go executor.serveCreateFuncServices()
return executor
return executor, nil
}
// All non-cached function service requests go through this goroutine
@@ -113,8 +111,18 @@ func (executor *Executor) serveCreateFuncServices() {
// still can serve other subsequent requests.
buffer := 10 // add some buffer time for specialization
specializationTimeout := req.function.Spec.InvokeStrategy.ExecutionStrategy.SpecializationTimeout
// set minimum specialization timeout to avoid illegal input and
// compatibility problem when applying old spec file that doesn't
// have specialization timeout field.
if specializationTimeout < fv1.DefaultSpecializationTimeOut {
specializationTimeout = fv1.DefaultSpecializationTimeOut
}
fnSpecializationTimeoutContext, cancel := context.WithTimeout(context.Background(),
time.Duration(req.function.Spec.InvokeStrategy.ExecutionStrategy.SpecializationTimeout+buffer)*time.Second)
time.Duration(specializationTimeout+buffer)*time.Second)
defer cancel()
fsvc, err := executor.createServiceForFunction(fnSpecializationTimeoutContext, req.function)
req.respChan <- &createFuncServiceResponse{
@@ -122,8 +130,6 @@ func (executor *Executor) serveCreateFuncServices() {
err: err,
}
delete(executor.fsCreateWg, crd.CacheKey(fnMetadata))
cancel()
wg.Done()
}()
} else {
@@ -134,7 +140,7 @@ func (executor *Executor) serveCreateFuncServices() {
wg.Wait()
// get the function service from the cache
fsvc, err := executor.fsCache.GetByFunction(fnMetadata)
fsvc, err := executor.getFunctionServiceFromCache(req.function)
// fsCache return error when the entry does not exist/expire.
// It normally happened if there are multiple requests are
@@ -155,18 +161,13 @@ func (executor *Executor) createServiceForFunction(ctx context.Context, fn *fv1.
zap.String("function_name", fn.Metadata.Name),
zap.String("function_namespace", fn.Metadata.Namespace))
executorType := fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType
var fsvc *fscache.FuncSvc
var fsvcErr error
switch executorType {
case fv1.ExecutorTypeNewdeploy:
fsvc, fsvcErr = executor.ndm.GetFuncSvc(ctx, fn)
default:
fsvc, fsvcErr = executor.gpm.GetFuncSvc(ctx, fn)
t := fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType
e, ok := executor.executorTypes[t]
if !ok {
return nil, errors.Errorf("Unknown executor type '%v'", t)
}
fsvc, fsvcErr := e.GetFuncSvc(ctx, fn)
if fsvcErr != nil {
e := "error creating service for function"
executor.logger.Error(e,
@@ -174,25 +175,18 @@ func (executor *Executor) createServiceForFunction(ctx context.Context, fn *fv1.
zap.String("function_name", fn.Metadata.Name),
zap.String("function_namespace", fn.Metadata.Namespace))
fsvcErr = errors.Wrap(fsvcErr, fmt.Sprintf("[%s] %s", fn.Metadata.Name, e))
} else if fsvc != nil {
_, err := executor.fsCache.Add(*fsvc)
if err != nil {
return nil, err
}
}
executor.fsCache.IncreaseColdStarts(fn.Metadata.Name, string(fn.Metadata.UID))
return fsvc, fsvcErr
}
// isValidAddress invokes isValidService or isValidPod depending on the type of executor
func (executor *Executor) isValidAddress(fsvc *fscache.FuncSvc) bool {
if fsvc.Executor == fscache.NEWDEPLOY {
return executor.ndm.IsValid(fsvc)
} else {
return executor.gpm.IsValid(fsvc)
func (executor *Executor) getFunctionServiceFromCache(fn *fv1.Function) (*fscache.FuncSvc, error) {
t := fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType
e, ok := executor.executorTypes[t]
if !ok {
return nil, errors.Errorf("Unknown executor type '%v'", t)
}
return e.GetFuncSvcFromCache(fn)
}
func serveMetric(logger *zap.Logger) {
@@ -223,26 +217,45 @@ func StartExecutor(logger *zap.Logger, functionNamespace string, envBuilderNames
}
restClient := fissionClient.GetCrdClient()
fsCache := fscache.MakeFunctionServiceCache(logger)
executorInstanceID := strings.ToLower(uniuri.NewLen(8))
poolID := strings.ToLower(uniuri.NewLen(8))
reaper.CleanupOldExecutorObjects(logger, kubernetesClient, poolID)
go reaper.CleanupRoleBindings(logger, kubernetesClient, fissionClient, functionNamespace, envBuilderNamespace, time.Minute*30)
logger.Info("Starting executor", zap.String("instanceID", executorInstanceID))
gpm := poolmgr.MakeGenericPoolManager(
logger,
fissionClient, kubernetesClient,
functionNamespace, fetcherConfig, poolID)
functionNamespace, fetcherConfig, executorInstanceID)
ndm := newdeploy.MakeNewDeploy(
logger,
fissionClient, kubernetesClient, restClient,
functionNamespace, fetcherConfig, poolID)
functionNamespace, fetcherConfig, executorInstanceID)
cms := cms.MakeConfigSecretController(logger, fissionClient, kubernetesClient, ndm, gpm)
executorTypes := make(map[fv1.ExecutorType]executortype.ExecutorType)
executorTypes[gpm.GetTypeName()] = gpm
executorTypes[ndm.GetTypeName()] = ndm
api := MakeExecutor(logger, gpm, ndm, cms, fissionClient, fsCache)
wg := &sync.WaitGroup{}
for _, et := range executorTypes {
wg.Add(1)
go func(et executortype.ExecutorType) {
defer wg.Done()
et.AdoptOrphanResources()
et.Run(context.Background())
}(et)
}
wg.Wait()
cms := cms.MakeConfigSecretController(logger, fissionClient, kubernetesClient, executorTypes)
cms.Run(context.Background())
api, err := MakeExecutor(logger, cms, fissionClient, executorTypes)
if err != nil {
return err
}
go reaper.CleanupRoleBindings(logger, kubernetesClient, fissionClient, functionNamespace, envBuilderNamespace, time.Minute*30)
go reaper.CleanupOldExecutorObjects(logger, kubernetesClient, executorInstanceID)
go api.Serve(port)
go serveMetric(logger)
+56
View File
@@ -0,0 +1,56 @@
/*
Copyright 2019 The Fission Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package executortype
import (
"context"
"go.uber.org/zap"
fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
"github.com/fission/fission/pkg/executor/fscache"
)
type ExecutorType interface {
// Run runs background job.
Run(context.Context)
// GetTypeName returns the name of executor type
GetTypeName() fv1.ExecutorType
// GetFuncSvc specializes function pod(s) and returns a service URL for the function.
GetFuncSvc(context.Context, *fv1.Function) (*fscache.FuncSvc, error)
// GetFuncSvcFromCache retrieves function service from cache.
GetFuncSvcFromCache(*fv1.Function) (*fscache.FuncSvc, error)
// DeleteFuncSvcFromCache deletes function service entry in cache.
DeleteFuncSvcFromCache(*fscache.FuncSvc)
// TapService updates the access time of function service entry to
// avoid idle pod reaper recycles pods.
TapService(serviceUrl string) error
// IsValid returns true if a function service is valid. Different executor types
// use distinct ways to examine the function service.
IsValid(*fscache.FuncSvc) bool
// RefreshFuncPods refreshes function pods if the secrets/configmaps pods reference to get updated.
RefreshFuncPods(*zap.Logger, fv1.Function) error
// AdoptOrphanResources adopts existing resources created by the deleted executor.
AdoptOrphanResources()
}
@@ -17,11 +17,11 @@ limitations under the License.
package newdeploy
import (
"errors"
"fmt"
"time"
multierror "github.com/hashicorp/go-multierror"
"github.com/pkg/errors"
"go.uber.org/zap"
appsv1 "k8s.io/api/apps/v1"
asv1 "k8s.io/api/autoscaling/v1"
@@ -29,6 +29,7 @@ import (
k8s_err "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
k8sTypes "k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/intstr"
fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
@@ -43,7 +44,7 @@ const (
)
func (deploy *NewDeploy) createOrGetDeployment(fn *fv1.Function, env *fv1.Environment,
deployName string, deployLabels map[string]string, deployNamespace string, firstcreate bool) (*appsv1.Deployment, error) {
deployName string, deployLabels map[string]string, deployAnnotations map[string]string, deployNamespace string, firstcreate bool) (*appsv1.Deployment, error) {
minScale := int32(fn.Spec.InvokeStrategy.ExecutionStrategy.MinScale)
specializationTimeout := int(fn.Spec.InvokeStrategy.ExecutionStrategy.SpecializationTimeout)
@@ -58,13 +59,23 @@ func (deploy *NewDeploy) createOrGetDeployment(fn *fv1.Function, env *fv1.Enviro
existingDepl, err := deploy.kubernetesClient.AppsV1().Deployments(deployNamespace).Get(deployName, metav1.GetOptions{})
if err == nil {
if waitForDeploy {
err = deploy.scaleDeployment(existingDepl.Namespace, existingDepl.Name, minScale)
if existingDepl.Labels[fv1.EXECUTOR_INSTANCEID_LABEL] != deploy.instanceID {
patch := fmt.Sprintf(`{"metadata":{"annotations":{"%v":"%v"}}}`, types.EXECUTOR_INSTANCEID_LABEL, deploy.instanceID)
existingDepl, err = deploy.kubernetesClient.AppsV1().Deployments(deployNamespace).Patch(deployName, k8sTypes.StrategicMergePatchType, []byte(patch))
if err != nil {
deploy.logger.Error("error scaling up function deployment", zap.Error(err), zap.String("function", fn.Metadata.Name))
deploy.logger.Warn("error patching executor instance ID of deploy", zap.Error(err),
zap.String("deploy", deployName), zap.String("ns", deployNamespace))
return nil, err
}
}
if waitForDeploy {
if *existingDepl.Spec.Replicas < minScale {
err = deploy.scaleDeployment(existingDepl.Namespace, existingDepl.Name, minScale)
if err != nil {
deploy.logger.Error("error scaling up function deployment", zap.Error(err), zap.String("function", fn.Metadata.Name))
return nil, err
}
}
if existingDepl.Status.AvailableReplicas < minScale {
existingDepl, err = deploy.waitForDeploy(existingDepl, minScale, specializationTimeout)
}
@@ -76,7 +87,7 @@ func (deploy *NewDeploy) createOrGetDeployment(fn *fv1.Function, env *fv1.Enviro
return nil, err
}
deployment, err := deploy.getDeploymentSpec(fn, env, deployName, deployLabels)
deployment, err := deploy.getDeploymentSpec(fn, env, deployName, deployNamespace, deployLabels, deployAnnotations)
if err != nil {
return nil, err
}
@@ -158,7 +169,7 @@ func (deploy *NewDeploy) deleteDeployment(ns string, name string) error {
}
func (deploy *NewDeploy) getDeploymentSpec(fn *fv1.Function, env *fv1.Environment,
deployName string, deployLabels map[string]string) (*appsv1.Deployment, error) {
deployName string, deployNamespace string, deployLabels map[string]string, deployAnnotations map[string]string) (*appsv1.Deployment, error) {
replicas := int32(fn.Spec.InvokeStrategy.ExecutionStrategy.MinScale)
@@ -174,6 +185,9 @@ func (deploy *NewDeploy) getDeploymentSpec(fn *fv1.Function, env *fv1.Environmen
if deploy.useIstio && env.Spec.AllowAccessToExternalNetwork {
podAnnotations["sidecar.istio.io/inject"] = "false"
}
for k, v := range deployAnnotations {
podAnnotations[k] = v
}
resources := deploy.getResources(env, fn)
// Set maxUnavailable and maxSurge to 20% is because we want
@@ -227,27 +241,32 @@ func (deploy *NewDeploy) getDeploymentSpec(fn *fv1.Function, env *fv1.Environmen
return nil, err
}
pod := apiv1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: deployLabels,
Annotations: podAnnotations,
},
Spec: apiv1.PodSpec{
Containers: []apiv1.Container{*container},
ServiceAccountName: "fission-fetcher",
TerminationGracePeriodSeconds: &gracePeriodSeconds,
},
}
pod.Spec = *(util.ApplyImagePullSecret(env.Spec.ImagePullSecret, pod.Spec))
deployment := &appsv1.Deployment{
ObjectMeta: metav1.ObjectMeta{
Name: deployName,
Labels: deployLabels,
Name: deployName,
Labels: deployLabels,
Annotations: deployAnnotations,
},
Spec: appsv1.DeploymentSpec{
Replicas: &replicas,
Selector: &metav1.LabelSelector{
MatchLabels: deployLabels,
},
Template: apiv1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: deployLabels,
Annotations: podAnnotations,
},
Spec: apiv1.PodSpec{
Containers: []apiv1.Container{*container},
ServiceAccountName: "fission-fetcher",
TerminationGracePeriodSeconds: &gracePeriodSeconds,
},
},
Template: pod,
Strategy: appsv1.DeploymentStrategy{
Type: appsv1.RollingUpdateDeploymentStrategyType,
RollingUpdate: &appsv1.RollingUpdateDeployment{
@@ -315,7 +334,10 @@ func (deploy *NewDeploy) getResources(env *fv1.Environment, fn *fv1.Function) ap
return resources
}
func (deploy *NewDeploy) createOrGetHpa(hpaName string, execStrategy *fv1.ExecutionStrategy, depl *appsv1.Deployment) (*asv1.HorizontalPodAutoscaler, error) {
func (deploy *NewDeploy) createOrGetHpa(hpaName string, execStrategy *fv1.ExecutionStrategy, depl *appsv1.Deployment, deployLabels map[string]string, deployAnnotations map[string]string) (*asv1.HorizontalPodAutoscaler, error) {
if depl == nil {
return nil, errors.New("failed to create HPA, found empty deployment")
}
minRepl := int32(execStrategy.MinScale)
if minRepl == 0 {
@@ -329,18 +351,22 @@ func (deploy *NewDeploy) createOrGetHpa(hpaName string, execStrategy *fv1.Execut
existingHpa, err := deploy.kubernetesClient.AutoscalingV1().HorizontalPodAutoscalers(depl.ObjectMeta.Namespace).Get(hpaName, metav1.GetOptions{})
if err == nil {
if existingHpa.Labels[fv1.EXECUTOR_INSTANCEID_LABEL] != deploy.instanceID {
patch := fmt.Sprintf(`{"metadata":{"annotations":{"%v":"%v"}}}`, types.EXECUTOR_INSTANCEID_LABEL, deploy.instanceID)
existingHpa, err = deploy.kubernetesClient.AutoscalingV1().HorizontalPodAutoscalers(depl.ObjectMeta.Namespace).Patch(hpaName, k8sTypes.StrategicMergePatchType, []byte(patch))
if err != nil {
deploy.logger.Warn("error patching executor instance ID of HPA", zap.Error(err),
zap.String("HPA", hpaName), zap.String("ns", depl.ObjectMeta.Namespace))
return nil, err
}
}
return existingHpa, err
}
if depl == nil {
return nil, errors.New("failed to create HPA, found empty deployment")
}
if err != nil && k8s_err.IsNotFound(err) {
} else if k8s_err.IsNotFound(err) {
hpa := asv1.HorizontalPodAutoscaler{
ObjectMeta: metav1.ObjectMeta{
Name: hpaName,
Labels: depl.Labels,
Name: hpaName,
Labels: deployLabels,
Annotations: deployAnnotations,
},
Spec: asv1.HorizontalPodAutoscalerSpec{
ScaleTargetRef: asv1.CrossVersionObjectReference{
@@ -378,15 +404,25 @@ func (deploy *NewDeploy) deleteHpa(ns string, name string) error {
return deploy.kubernetesClient.AutoscalingV1().HorizontalPodAutoscalers(ns).Delete(name, &metav1.DeleteOptions{})
}
func (deploy *NewDeploy) createOrGetSvc(deployLabels map[string]string, svcName string, svcNamespace string) (*apiv1.Service, error) {
func (deploy *NewDeploy) createOrGetSvc(deployLabels map[string]string, deployAnnotations map[string]string, svcName string, svcNamespace string) (*apiv1.Service, error) {
existingSvc, err := deploy.kubernetesClient.CoreV1().Services(svcNamespace).Get(svcName, metav1.GetOptions{})
if err == nil {
if existingSvc.Labels[fv1.EXECUTOR_INSTANCEID_LABEL] != deploy.instanceID {
patch := fmt.Sprintf(`{"metadata":{"annotations":{"%v":"%v"}}}`, types.EXECUTOR_INSTANCEID_LABEL, deploy.instanceID)
existingSvc, err = deploy.kubernetesClient.CoreV1().Services(svcNamespace).Patch(svcName, k8sTypes.StrategicMergePatchType, []byte(patch))
if err != nil {
deploy.logger.Warn("error patching executor instance ID of service", zap.Error(err),
zap.String("service", svcName), zap.String("ns", svcNamespace))
return nil, err
}
}
return existingSvc, err
} else if k8s_err.IsNotFound(err) {
service := &apiv1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: svcName,
Labels: deployLabels,
Name: svcName,
Labels: deployLabels,
Annotations: deployAnnotations,
},
Spec: apiv1.ServiceSpec{
Ports: []apiv1.ServicePort{
@@ -19,14 +19,13 @@ package newdeploy
import (
"context"
"fmt"
"math/rand"
"os"
"strconv"
"strings"
"sync"
"time"
"github.com/dchest/uniuri"
"github.com/fission/fission/pkg/throttler"
"github.com/fission/fission/pkg/utils"
multierror "github.com/hashicorp/go-multierror"
"github.com/pkg/errors"
"go.uber.org/zap"
@@ -43,11 +42,16 @@ import (
fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
"github.com/fission/fission/pkg/crd"
"github.com/fission/fission/pkg/executor/executortype"
"github.com/fission/fission/pkg/executor/fscache"
fetcherConfig "github.com/fission/fission/pkg/fetcher/config"
"github.com/fission/fission/pkg/throttler"
"github.com/fission/fission/pkg/types"
"github.com/fission/fission/pkg/utils"
)
var _ executortype.ExecutorType = &NewDeploy{}
type (
NewDeploy struct {
logger *zap.Logger
@@ -83,7 +87,7 @@ func MakeNewDeploy(
namespace string,
fetcherConfig *fetcherConfig.Config,
instanceID string,
) *NewDeploy {
) executortype.ExecutorType {
enableIstio := false
if len(os.Getenv("ENABLE_ISTIO")) > 0 {
istio, err := strconv.ParseBool(os.Getenv("ENABLE_ISTIO"))
@@ -131,6 +135,184 @@ func (deploy *NewDeploy) Run(ctx context.Context) {
go deploy.idleObjectReaper()
}
func (deploy *NewDeploy) GetTypeName() fv1.ExecutorType {
return fv1.ExecutorTypeNewdeploy
}
func (deploy *NewDeploy) GetFuncSvc(ctx context.Context, fn *fv1.Function) (*fscache.FuncSvc, error) {
// TODO: client-go doesn't support to pass in context.
// Once it supports context, we should change the signature of method.
// https://github.com/kubernetes/kubernetes/issues/46503
return deploy.createFunction(fn, false)
}
func (deploy *NewDeploy) GetFuncSvcFromCache(fn *fv1.Function) (*fscache.FuncSvc, error) {
return deploy.fsCache.GetByFunction(&fn.Metadata)
}
func (deploy *NewDeploy) DeleteFuncSvcFromCache(fsvc *fscache.FuncSvc) {
deploy.fsCache.DeleteEntry(fsvc)
}
func (deploy *NewDeploy) TapService(svcHost string) error {
err := deploy.fsCache.TouchByAddress(svcHost)
if err != nil {
return err
}
return nil
}
// IsValid does a get on the service address to ensure it's a valid service, then
// scale deployment to 1 replica if there are no available replicas for function.
// Return true if no error occurs, return false otherwise.
func (deploy *NewDeploy) IsValid(fsvc *fscache.FuncSvc) bool {
service := strings.Split(fsvc.Address, ".")
if len(service) == 0 {
return false
}
_, err := deploy.kubernetesClient.CoreV1().Services(service[1]).Get(service[0], metav1.GetOptions{})
if err != nil {
if !k8sErrs.IsNotFound(err) {
deploy.logger.Error("error validating function service address", zap.String("function", fsvc.Function.Name), zap.Error(err))
}
return false
}
deployObj := getDeploymentObj(fsvc.KubernetesObjects)
if deployObj == nil {
deploy.logger.Error("deployment obj for function does not exist", zap.String("function", fsvc.Function.Name))
return false
}
currentDeploy, err := deploy.kubernetesClient.AppsV1().
Deployments(deployObj.Namespace).Get(deployObj.Name, metav1.GetOptions{})
if err != nil {
if !k8sErrs.IsNotFound(err) {
deploy.logger.Error("error validating function deployment", zap.Error(err), zap.String("function", fsvc.Function.Name))
}
return false
}
// return directly when available replicas > 0
if currentDeploy.Status.AvailableReplicas > 0 {
return true
}
return false
}
// RefreshFuncPods deleted pods related to the function so that new pods are replenished
func (deploy *NewDeploy) RefreshFuncPods(logger *zap.Logger, f fv1.Function) error {
env, err := deploy.fissionClient.Environments(f.Spec.Environment.Namespace).Get(f.Spec.Environment.Name)
if err != nil {
return err
}
funcLabels := deploy.getDeployLabels(f.Metadata, metav1.ObjectMeta{
Name: f.Spec.Environment.Name,
Namespace: f.Spec.Environment.Namespace,
UID: env.Metadata.UID,
})
dep, err := deploy.kubernetesClient.AppsV1().Deployments(metav1.NamespaceAll).List(metav1.ListOptions{
LabelSelector: labels.Set(funcLabels).AsSelector().String(),
})
if err != nil {
return err
}
patch := fmt.Sprintf(`{"spec" : {"template": {"spec":{"containers":[{"name": "%s", "env":[{"name": "%s", "value": "%s"}]}]}}}}`,
f.Metadata.Name,
fv1.LastUpdateTimestamp,
time.Now().String())
// Ideally there should be only one deployment but for now we rely on label/selector to ensure that condition
for _, deployment := range dep.Items {
_, err := deploy.kubernetesClient.AppsV1().Deployments(deployment.ObjectMeta.Namespace).Patch(deployment.ObjectMeta.Name,
k8sTypes.StrategicMergePatchType,
[]byte(patch))
if err != nil {
return err
}
}
return nil
}
func (deploy *NewDeploy) AdoptOrphanResources() {
l := map[string]string{
types.EXECUTOR_TYPE: string(fv1.ExecutorTypeNewdeploy),
}
podList, err := deploy.kubernetesClient.CoreV1().Pods(metav1.NamespaceAll).List(metav1.ListOptions{
LabelSelector: labels.Set(l).AsSelector().String(),
})
if err != nil {
deploy.logger.Error("error getting pod list", zap.Error(err))
return
}
podWG := &sync.WaitGroup{}
for i := range podList.Items {
pod := &podList.Items[i]
if !utils.IsReadyPod(pod) {
continue
}
podWG.Add(1)
go func() {
defer podWG.Done()
// avoid too many requests arrive Kubernetes API server at the same time.
time.Sleep(time.Duration(rand.Intn(30)) * time.Millisecond)
patch := fmt.Sprintf(`{"metadata":{"annotations":{"%v":"%v"}}}`, types.EXECUTOR_INSTANCEID_LABEL, deploy.instanceID)
pod, err = deploy.kubernetesClient.CoreV1().Pods(pod.Namespace).Patch(pod.Name, k8sTypes.StrategicMergePatchType, []byte(patch))
if err != nil {
// just log the error since it won't affect the function serving
deploy.logger.Warn("error patching executor instance ID of pod", zap.Error(err),
zap.String("pod", pod.Name), zap.String("ns", pod.Namespace))
return
}
deploy.logger.Info("adopt newdeploy function pod",
zap.String("pod", pod.Name), zap.Any("labels", pod.Labels), zap.Any("annotations", pod.Annotations))
}()
}
fnList, err := deploy.fissionClient.Functions(metav1.NamespaceAll).List(metav1.ListOptions{})
if err != nil {
deploy.logger.Error("error getting function list", zap.Error(err))
return
}
deployWG := &sync.WaitGroup{}
for i := range fnList.Items {
fn := &fnList.Items[i]
if fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType == fv1.ExecutorTypeNewdeploy {
deployWG.Add(1)
go func() {
defer deployWG.Done()
_, err = deploy.fnCreate(fn, true)
if err != nil {
deploy.logger.Warn("failed to adopt resources for function", zap.Error(err))
return
}
deploy.logger.Info("adopt resources for function", zap.String("function", fn.Metadata.Name))
}()
}
}
podWG.Wait()
deployWG.Wait()
}
func (deploy *NewDeploy) initFuncController() (k8sCache.Store, k8sCache.Controller) {
resyncPeriod := 30 * time.Second
listWatch := k8sCache.NewListWatchFromClient(deploy.crdClient, "functions", metav1.NamespaceAll, fields.Everything())
@@ -224,49 +406,6 @@ func (deploy *NewDeploy) getEnvFunctions(m *metav1.ObjectMeta) []fv1.Function {
return relatedFunctions
}
func (deploy *NewDeploy) GetFuncSvc(ctx context.Context, fn *fv1.Function) (*fscache.FuncSvc, error) {
return deploy.createFunction(fn, false)
}
// RefreshFuncPods deleted pods related to the function so that new pods are replenished
func (deploy *NewDeploy) RefreshFuncPods(logger *zap.Logger, f fv1.Function) error {
env, err := deploy.fissionClient.Environments(f.Spec.Environment.Namespace).Get(f.Spec.Environment.Name)
if err != nil {
return err
}
funcLabels := deploy.getDeployLabels(f.Metadata, metav1.ObjectMeta{
Name: f.Spec.Environment.Name,
Namespace: f.Spec.Environment.Namespace,
UID: env.Metadata.UID,
})
dep, err := deploy.kubernetesClient.AppsV1().Deployments(metav1.NamespaceAll).List(metav1.ListOptions{
LabelSelector: labels.Set(funcLabels).AsSelector().String(),
})
if err != nil {
return err
}
patch := fmt.Sprintf(`{"spec" : {"template": {"spec":{"containers":[{"name": "%s", "env":[{"name": "%s", "value": "%s"}]}]}}}}`,
f.Metadata.Name,
fv1.LastUpdateTimestamp,
time.Now().String())
// Ideally there should be only one deployment but for now we rely on label/selector to ensure that condition
for _, deployment := range dep.Items {
_, err := deploy.kubernetesClient.AppsV1().Deployments(deployment.ObjectMeta.Namespace).Patch(deployment.ObjectMeta.Name,
k8sTypes.StrategicMergePatchType,
[]byte(patch))
if err != nil {
return err
}
}
return nil
}
func (deploy *NewDeploy) createFunction(fn *fv1.Function, firstcreate bool) (*fscache.FuncSvc, error) {
if fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType != fv1.ExecutorTypeNewdeploy {
return nil, nil
@@ -316,15 +455,8 @@ func (deploy *NewDeploy) fnCreate(fn *fv1.Function, firstcreate bool) (*fscache.
}
objName := deploy.getObjName(fn)
if !firstcreate {
// retrieve back the previous obj name for later use.
fsvc, err := deploy.fsCache.GetByFunctionUID(fn.Metadata.UID)
if err != nil {
return nil, errors.Wrap(err, "error getting existed function service cache")
}
objName = fsvc.Name
}
deployLabels := deploy.getDeployLabels(fn.Metadata, env.Metadata)
deployAnnotations := deploy.getDeployAnnotations(fn.Metadata)
// to support backward compatibility, if the function was created in default ns, we fall back to creating the
// deployment of the function in fission-function ns
@@ -338,7 +470,7 @@ func (deploy *NewDeploy) fnCreate(fn *fv1.Function, firstcreate bool) (*fscache.
// Since newdeploy waits for pods of deployment to be ready,
// change the order of kubeObject creation (create service first,
// then deployment) to take advantage of waiting time.
svc, err := deploy.createOrGetSvc(deployLabels, objName, ns)
svc, err := deploy.createOrGetSvc(deployLabels, deployAnnotations, objName, ns)
if err != nil {
deploy.logger.Error("error creating service", zap.Error(err), zap.String("service", objName))
go deploy.cleanupNewdeploy(ns, objName)
@@ -346,14 +478,14 @@ func (deploy *NewDeploy) fnCreate(fn *fv1.Function, firstcreate bool) (*fscache.
}
svcAddress := fmt.Sprintf("%v.%v", svc.Name, svc.Namespace)
depl, err := deploy.createOrGetDeployment(fn, env, objName, deployLabels, ns, firstcreate)
depl, err := deploy.createOrGetDeployment(fn, env, objName, deployLabels, deployAnnotations, ns, firstcreate)
if err != nil {
deploy.logger.Error("error creating deployment", zap.Error(err), zap.String("deployment", objName))
go deploy.cleanupNewdeploy(ns, objName)
return nil, errors.Wrapf(err, "error creating deployment %v", objName)
}
hpa, err := deploy.createOrGetHpa(objName, &fn.Spec.InvokeStrategy.ExecutionStrategy, depl)
hpa, err := deploy.createOrGetHpa(objName, &fn.Spec.InvokeStrategy.ExecutionStrategy, depl, deployLabels, deployAnnotations)
if err != nil {
deploy.logger.Error("error creating HPA", zap.Error(err), zap.String("hpa", objName))
go deploy.cleanupNewdeploy(ns, objName)
@@ -394,7 +526,7 @@ func (deploy *NewDeploy) fnCreate(fn *fv1.Function, firstcreate bool) (*fscache.
Environment: env,
Address: svcAddress,
KubernetesObjects: kubeObjRefs,
Executor: fscache.NEWDEPLOY,
Executor: fv1.ExecutorTypeNewdeploy,
}
_, err = deploy.fsCache.Add(*fsvc)
@@ -402,6 +534,9 @@ func (deploy *NewDeploy) fnCreate(fn *fv1.Function, firstcreate bool) (*fscache.
deploy.logger.Error("error adding function to cache", zap.Error(err), zap.Any("function", fsvc.Function))
return fsvc, err
}
deploy.fsCache.IncreaseColdStarts(fn.Metadata.Name, string(fn.Metadata.UID))
return fsvc, nil
}
@@ -544,12 +679,6 @@ func (deploy *NewDeploy) updateFuncDeployment(fn *fv1.Function, env *fv1.Environ
deploy.logger.Info("updating deployment due to function/environment update",
zap.String("deployment", fnObjName), zap.Any("function", fn.Metadata.Name))
newDeployment, err := deploy.getDeploymentSpec(fn, env, fnObjName, deployLabels)
if err != nil {
deploy.updateStatus(fn, err, "failed to get new deployment spec while updating function")
return err
}
// to support backward compatibility, if the function was created in default ns, we fall back to creating the
// deployment of the function in fission-function ns
ns := deploy.namespace
@@ -557,6 +686,12 @@ func (deploy *NewDeploy) updateFuncDeployment(fn *fv1.Function, env *fv1.Environ
ns = fn.Metadata.Namespace
}
newDeployment, err := deploy.getDeploymentSpec(fn, env, fnObjName, ns, deployLabels, deploy.getDeployAnnotations(fn.Metadata))
if err != nil {
deploy.updateStatus(fn, err, "failed to get new deployment spec while updating function")
return err
}
err = deploy.updateDeployment(newDeployment, ns)
if err != nil {
deploy.updateStatus(fn, err, "failed to update deployment while updating function")
@@ -603,19 +738,27 @@ func (deploy *NewDeploy) fnDelete(fn *fv1.Function) error {
// getObjName returns a unique name for kubernetes objects of function
func (deploy *NewDeploy) getObjName(fn *fv1.Function) string {
return strings.ToLower(fmt.Sprintf("newdeploy-%v-%v-%v", fn.Metadata.Name, fn.Metadata.Namespace, uniuri.NewLen(8)))
// use meta uuid of function, this ensure we always get the same name for the same function.
uid := fn.Metadata.UID[len(fn.Metadata.UID)-17:]
return strings.ToLower(fmt.Sprintf("newdeploy-%v-%v-%v", fn.Metadata.Name, fn.Metadata.Namespace, uid))
}
func (deploy *NewDeploy) getDeployLabels(fnMeta metav1.ObjectMeta, envMeta metav1.ObjectMeta) map[string]string {
return map[string]string{
types.EXECUTOR_TYPE: string(fv1.ExecutorTypeNewdeploy),
types.ENVIRONMENT_NAME: envMeta.Name,
types.ENVIRONMENT_NAMESPACE: envMeta.Namespace,
types.ENVIRONMENT_UID: string(envMeta.UID),
types.FUNCTION_NAME: fnMeta.Name,
types.FUNCTION_NAMESPACE: fnMeta.Namespace,
types.FUNCTION_UID: string(fnMeta.UID),
}
}
func (deploy *NewDeploy) getDeployAnnotations(fnMeta metav1.ObjectMeta) map[string]string {
return map[string]string{
types.EXECUTOR_INSTANCEID_LABEL: deploy.instanceID,
types.EXECUTOR_TYPE: fv1.ExecutorTypeNewdeploy,
types.ENVIRONMENT_NAME: envMeta.Name,
types.ENVIRONMENT_NAMESPACE: envMeta.Namespace,
types.ENVIRONMENT_UID: string(envMeta.UID),
types.FUNCTION_NAME: fnMeta.Name,
types.FUNCTION_NAMESPACE: fnMeta.Namespace,
types.FUNCTION_UID: string(fnMeta.UID),
types.FUNCTION_RESOURCE_VERSION: fnMeta.ResourceVersion,
}
}
@@ -625,42 +768,6 @@ func (deploy *NewDeploy) updateStatus(fn *fv1.Function, err error, message strin
deploy.logger.Error("function status update", zap.Error(err), zap.Any("function", fn), zap.String("message", message))
}
// IsValid does a get on the service address to ensure it's a valid service, then
// scale deployment to 1 replica if there are no available replicas for function.
// Return true if no error occurs, return false otherwise.
func (deploy *NewDeploy) IsValid(fsvc *fscache.FuncSvc) bool {
service := strings.Split(fsvc.Address, ".")
if len(service) == 0 {
return false
}
_, err := deploy.kubernetesClient.CoreV1().Services(service[1]).Get(service[0], metav1.GetOptions{})
if err != nil {
deploy.logger.Error("error validating function service address", zap.String("function", fsvc.Function.Name), zap.Error(err))
return false
}
deployObj := getDeploymentObj(fsvc.KubernetesObjects)
if deployObj == nil {
deploy.logger.Error("deployment obj for function does not exist", zap.String("function", fsvc.Function.Name))
return false
}
currentDeploy, err := deploy.kubernetesClient.AppsV1().
Deployments(deployObj.Namespace).Get(deployObj.Name, metav1.GetOptions{})
if err != nil {
deploy.logger.Error("error validating function deployment", zap.Error(err), zap.String("function", fsvc.Function.Name))
return false
}
// return directly when available replicas > 0
if currentDeploy.Status.AvailableReplicas > 0 {
return true
}
return false
}
// idleObjectReaper reaps objects after certain idle time
func (deploy *NewDeploy) idleObjectReaper() {
@@ -685,7 +792,7 @@ func (deploy *NewDeploy) idleObjectReaper() {
}
for _, fsvc := range funcSvcs {
if fsvc.Executor != fscache.NEWDEPLOY {
if fsvc.Executor != fv1.ExecutorTypeNewdeploy {
continue
}
@@ -701,7 +808,7 @@ func (deploy *NewDeploy) idleObjectReaper() {
if err != nil {
// Newdeploy manager handles the function delete event and clean cache/kubeobjs itself,
// so we ignore the not found error for functions with newdeploy executor type here.
if k8sErrs.IsNotFound(err) && fsvc.Executor == fscache.NEWDEPLOY {
if k8sErrs.IsNotFound(err) && fsvc.Executor == fv1.ExecutorTypeNewdeploy {
continue
}
deploy.logger.Error("error getting function", zap.Error(err), zap.String("function", fsvc.Function.Name))
@@ -717,7 +824,7 @@ func (deploy *NewDeploy) idleObjectReaper() {
currentDeploy, err := deploy.kubernetesClient.AppsV1().
Deployments(deployObj.Namespace).Get(deployObj.Name, metav1.GetOptions{})
if err != nil {
deploy.logger.Error("error validating function deployment", zap.Error(err), zap.String("function", fsvc.Function.Name))
deploy.logger.Error("error getting function deployment", zap.Error(err), zap.String("function", fsvc.Function.Name))
continue
}
@@ -33,8 +33,10 @@ import (
"go.uber.org/zap"
appsv1 "k8s.io/api/apps/v1"
apiv1 "k8s.io/api/core/v1"
k8sErrs "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
k8sTypes "k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/intstr"
"k8s.io/client-go/kubernetes"
@@ -64,9 +66,9 @@ type (
kubernetesClient *kubernetes.Clientset
fissionClient *crd.FissionClient
instanceId string // poolmgr instance id
labelsForPool map[string]string
requestChannel chan *choosePodRequest
fetcherConfig *fetcherConfig.Config
stopCh context.CancelFunc
}
// serialize the choosing of pods so that choices don't conflict
@@ -97,6 +99,8 @@ func MakeGenericPool(
gpLogger.Info("creating pool", zap.Any("environment", env.Metadata))
ctx, stopCh := context.WithCancel(context.Background())
// TODO: in general we need to provide the user a way to configure pools. Initial
// replicas, autoscaling params, various timeouts, etc.
gp := &GenericPool{
@@ -116,6 +120,7 @@ func MakeGenericPool(
instanceId: instanceId,
useSvc: false, // defaults off -- svc takes a second or more to become routable, slowing cold start
useIstio: enableIstio, // defaults off -- istio integration requires pod relabeling and it takes a second or more to become routable, slowing cold start
stopCh: stopCh,
}
gp.runtimeImagePullPolicy = utils.GetImagePullPolicy(os.Getenv("RUNTIME_IMAGE_PULL_POLICY"))
@@ -127,7 +132,7 @@ func MakeGenericPool(
}
// Labels for generic deployment/RS/pods.
gp.labelsForPool = gp.getDeployLabels()
//gp.labelsForPool = gp.getDeployLabels()
// create the pool
err = gp.createPool()
@@ -136,31 +141,41 @@ func MakeGenericPool(
}
gpLogger.Info("deployment created", zap.Any("environment", env.Metadata))
go gp.choosePodService()
go gp.choosePodService(ctx)
return gp, nil
}
func (gp *GenericPool) getDeployLabels() map[string]string {
return map[string]string{
types.EXECUTOR_TYPE: string(fv1.ExecutorTypePoolmgr),
types.ENVIRONMENT_NAME: gp.env.Metadata.Name,
types.ENVIRONMENT_NAMESPACE: gp.env.Metadata.Namespace,
types.ENVIRONMENT_UID: string(gp.env.Metadata.UID),
"managed": "true", // this allows us to easily find pods managed by the deployment
}
}
func (gp *GenericPool) getDeployAnnotations() map[string]string {
return map[string]string{
fv1.EXECUTOR_INSTANCEID_LABEL: gp.instanceId,
types.EXECUTOR_TYPE: fv1.ExecutorTypePoolmgr,
types.ENVIRONMENT_NAME: gp.env.Metadata.Name,
types.ENVIRONMENT_NAMESPACE: gp.env.Metadata.Namespace,
types.ENVIRONMENT_UID: string(gp.env.Metadata.UID),
"managed": "true", // this allows us to easily find pods managed by the deployment
}
}
// choosePodService serializes the choosing of pods
func (gp *GenericPool) choosePodService() {
for req := range gp.requestChannel {
pod, err := gp._choosePod(req.newLabels)
if err != nil {
req.responseChannel <- &choosePodResponse{error: err}
continue
func (gp *GenericPool) choosePodService(ctx context.Context) {
for {
select {
case req := <-gp.requestChannel:
pod, err := gp._choosePod(req.newLabels)
if err != nil {
req.responseChannel <- &choosePodResponse{error: err}
continue
}
req.responseChannel <- &choosePodResponse{pod: pod}
case <-ctx.Done():
return
}
req.responseChannel <- &choosePodResponse{pod: pod}
}
}
@@ -330,12 +345,15 @@ func (gp *GenericPool) specializePod(ctx context.Context, pod *apiv1.Pod, fn *fv
// getPoolName returns a unique name of an environment
func (gp *GenericPool) getPoolName() string {
return strings.ToLower(fmt.Sprintf("poolmgr-%v-%v-%v", gp.env.Metadata.Name, gp.env.Metadata.Namespace, uniuri.NewLen(8)))
return strings.ToLower(fmt.Sprintf("poolmgr-%v-%v-%v", gp.env.Metadata.Name, gp.env.Metadata.Namespace, gp.env.Metadata.ResourceVersion))
}
// A pool is a deployment of generic containers for an env. This
// creates the pool but doesn't wait for any pods to be ready.
func (gp *GenericPool) createPool() error {
deployLabels := gp.getDeployLabels()
deployAnnotations := gp.getDeployAnnotations()
// Use long terminationGracePeriodSeconds for connection draining in case that
// pod still runs user functions.
gracePeriodSeconds := int64(6 * 60)
@@ -350,6 +368,9 @@ func (gp *GenericPool) createPool() error {
if gp.useIstio && gp.env.Spec.AllowAccessToExternalNetwork {
podAnnotations["sidecar.istio.io/inject"] = "false"
}
for k, v := range deployAnnotations {
podAnnotations[k] = v
}
container, err := util.MergeContainer(&apiv1.Container{
Name: gp.env.Metadata.Name,
@@ -389,30 +410,35 @@ func (gp *GenericPool) createPool() error {
return err
}
pod := apiv1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: deployLabels,
Annotations: podAnnotations,
},
Spec: apiv1.PodSpec{
Containers: []apiv1.Container{*container},
ServiceAccountName: "fission-fetcher",
// TerminationGracePeriodSeconds should be equal to the
// sleep time of preStop to make sure that SIGTERM is sent
// to pod after 6 mins.
TerminationGracePeriodSeconds: &gracePeriodSeconds,
},
}
pod.Spec = *(util.ApplyImagePullSecret(gp.env.Spec.ImagePullSecret, pod.Spec))
deployment := &appsv1.Deployment{
ObjectMeta: metav1.ObjectMeta{
Name: gp.getPoolName(),
Labels: gp.labelsForPool,
Name: gp.getPoolName(),
Labels: deployLabels,
Annotations: deployAnnotations,
},
Spec: appsv1.DeploymentSpec{
Replicas: &gp.replicas,
Selector: &metav1.LabelSelector{
MatchLabels: gp.labelsForPool,
},
Template: apiv1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: gp.labelsForPool,
Annotations: podAnnotations,
},
Spec: apiv1.PodSpec{
Containers: []apiv1.Container{*container},
ServiceAccountName: "fission-fetcher",
// TerminationGracePeriodSeconds should be equal to the
// sleep time of preStop to make sure that SIGTERM is sent
// to pod after 6 mins.
TerminationGracePeriodSeconds: &gracePeriodSeconds,
},
MatchLabels: deployLabels,
},
Template: pod,
},
}
@@ -430,11 +456,25 @@ func (gp *GenericPool) createPool() error {
deployment.Spec.Template.Spec = *newPodSpec
}
_, err = gp.kubernetesClient.AppsV1().Deployments(gp.namespace).Get(deployment.Name, metav1.GetOptions{})
if err == nil {
patch := fmt.Sprintf(`{"metadata":{"annotations":{"%v":"%v"}}}`, fv1.EXECUTOR_INSTANCEID_LABEL, gp.instanceId)
depl, err := gp.kubernetesClient.AppsV1().Deployments(gp.namespace).Patch(deployment.Name, k8sTypes.StrategicMergePatchType, []byte(patch))
if err == nil {
gp.deployment = depl
return nil
}
} else if !k8sErrs.IsNotFound(err) {
gp.logger.Error("error getting deployment in kubernetes", zap.Error(err), zap.String("deployment", deployment.Name))
return err
}
depl, err := gp.kubernetesClient.AppsV1().Deployments(gp.namespace).Create(deployment)
if err != nil {
gp.logger.Error("error creating deployment in kubernetes", zap.Error(err), zap.String("deployment", deployment.Name))
return err
}
gp.deployment = depl
return nil
}
@@ -467,14 +507,17 @@ func (gp *GenericPool) waitForReadyPod() error {
// Since even single pod is not ready, choosing the first pod to inspect is a good approximation. In future this can be done better
pod := podList.Items[0]
multierr := &multierror.Error{}
errs := &multierror.Error{}
for _, cStatus := range pod.Status.ContainerStatuses {
if !cStatus.Ready {
multierr = multierror.Append(multierr, errors.New(fmt.Sprintf("%v: %v", cStatus.State.Waiting.Reason, cStatus.State.Waiting.Message)))
errs = multierror.Append(errs, errors.New(fmt.Sprintf("%v: %v", cStatus.State.Waiting.Reason, cStatus.State.Waiting.Message)))
}
}
return errors.Wrapf(multierr, "Timeout: waited too long for pod of deployment %v in namespace %v to be ready",
gp.deployment.ObjectMeta.Name, gp.namespace)
if errs.ErrorOrNil() != nil {
return errors.Wrapf(errs, "Timeout: waited too long for pod of deployment %v in namespace %v to be ready",
gp.deployment.ObjectMeta.Name, gp.namespace)
}
return nil
}
time.Sleep(1000 * time.Millisecond)
}
@@ -490,7 +533,7 @@ func (gp *GenericPool) createSvc(name string, labels map[string]string) (*apiv1.
Ports: []apiv1.ServicePort{
{
Protocol: apiv1.ProtocolTCP,
Port: 80,
Port: 8888,
TargetPort: intstr.FromInt(8888),
},
},
@@ -577,7 +620,7 @@ func (gp *GenericPool) getFuncSvc(ctx context.Context, fn *fv1.Function) (*fscac
// the fission router isn't in the same namespace, so return a
// namespace-qualified hostname
svcHost = fmt.Sprintf("%v.%v", svcName, gp.namespace)
svcHost = fmt.Sprintf("%v.%v:8888", svcName, gp.namespace)
} else if gp.useIstio {
svc := utils.GetFunctionIstioServiceName(fn.Metadata.Name, fn.Metadata.Namespace)
svcHost = fmt.Sprintf("%v.%v:8888", svc, gp.namespace)
@@ -585,6 +628,18 @@ func (gp *GenericPool) getFuncSvc(ctx context.Context, fn *fv1.Function) (*fscac
svcHost = fmt.Sprintf("%v:8888", pod.Status.PodIP)
}
// patch svc-host and resource version to the pod annotations for new executor to adopt the pod
patch := fmt.Sprintf(`{"metadata":{"annotations":{"%v":"%v","%v":"%v"}}}`,
types.ANNOTATION_SVC_HOST, svcHost, types.FUNCTION_RESOURCE_VERSION, fn.Metadata.ResourceVersion)
p, err := gp.kubernetesClient.CoreV1().Pods(pod.Namespace).Patch(pod.Name, k8sTypes.StrategicMergePatchType, []byte(patch))
if err != nil {
// just log the error since it won't affect the function serving
gp.logger.Warn("error patching svc-host to pod", zap.Error(err),
zap.String("pod", pod.Name), zap.String("ns", pod.Namespace))
} else {
pod = p
}
gp.logger.Info("specialized pod",
zap.String("pod", pod.ObjectMeta.Name),
zap.String("podNamespace", pod.ObjectMeta.Namespace),
@@ -610,7 +665,7 @@ func (gp *GenericPool) getFuncSvc(ctx context.Context, fn *fv1.Function) (*fscac
Environment: gp.env,
Address: svcHost,
KubernetesObjects: kubeObjRefs,
Executor: fscache.POOLMGR,
Executor: fv1.ExecutorTypePoolmgr,
Ctime: time.Now(),
Atime: time.Now(),
}
@@ -619,15 +674,21 @@ func (gp *GenericPool) getFuncSvc(ctx context.Context, fn *fv1.Function) (*fscac
if err != nil {
return nil, err
}
gp.fsCache.IncreaseColdStarts(fn.Metadata.Name, string(fn.Metadata.UID))
return fsvc, nil
}
// destroys the pool -- the deployment, replicaset and pods
func (gp *GenericPool) destroy() error {
gp.stopCh()
deletePropagation := metav1.DeletePropagationBackground
delOpt := metav1.DeleteOptions{
PropagationPolicy: &deletePropagation,
}
err := gp.kubernetesClient.AppsV1().
Deployments(gp.namespace).Delete(gp.deployment.ObjectMeta.Name, &delOpt)
if err != nil {
@@ -18,13 +18,16 @@ package poolmgr
import (
"context"
"fmt"
"math/rand"
"os"
"strconv"
"strings"
"sync"
"time"
"github.com/fission/fission/pkg/utils"
"go.uber.org/zap"
apiv1 "k8s.io/api/core/v1"
k8serrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
@@ -35,12 +38,16 @@ import (
fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
"github.com/fission/fission/pkg/cache"
"github.com/fission/fission/pkg/crd"
"github.com/fission/fission/pkg/executor/executortype"
"github.com/fission/fission/pkg/executor/fscache"
"github.com/fission/fission/pkg/executor/reaper"
fetcherConfig "github.com/fission/fission/pkg/fetcher/config"
"github.com/fission/fission/pkg/types"
"github.com/fission/fission/pkg/utils"
)
var _ executortype.ExecutorType = &GenericPoolManager{}
type requestType int
const (
@@ -90,7 +97,7 @@ func MakeGenericPoolManager(
kubernetesClient *kubernetes.Clientset,
functionNamespace string,
fetcherConfig *fetcherConfig.Config,
instanceId string) *GenericPoolManager {
instanceId string) executortype.ExecutorType {
gpmLogger := logger.Named("generic_pool_manager")
@@ -107,6 +114,7 @@ func MakeGenericPoolManager(
idlePodReapTime: 2 * time.Minute,
fetcherConfig: fetcherConfig,
}
go gpm.service()
go gpm.eagerPoolCreator()
@@ -132,6 +140,67 @@ func (gpm *GenericPoolManager) Run(ctx context.Context) {
go gpm.idleObjectReaper()
}
func (gpm *GenericPoolManager) GetTypeName() fv1.ExecutorType {
return fv1.ExecutorTypePoolmgr
}
func (gpm *GenericPoolManager) GetFuncSvc(ctx context.Context, fn *fv1.Function) (*fscache.FuncSvc, error) {
// from Func -> get Env
gpm.logger.Debug("getting environment for function", zap.String("function", fn.Metadata.Name))
env, err := gpm.getFunctionEnv(fn)
if err != nil {
return nil, err
}
pool, err := gpm.getPool(env)
if err != nil {
return nil, err
}
// from GenericPool -> get one function container
// (this also adds to the cache)
gpm.logger.Debug("getting function service from pool", zap.String("function", fn.Metadata.Name))
return pool.getFuncSvc(ctx, fn)
}
func (gpm *GenericPoolManager) GetFuncSvcFromCache(fn *fv1.Function) (*fscache.FuncSvc, error) {
return gpm.fsCache.GetByFunction(&fn.Metadata)
}
func (gpm *GenericPoolManager) DeleteFuncSvcFromCache(fsvc *fscache.FuncSvc) {
gpm.fsCache.DeleteEntry(fsvc)
}
func (gpm *GenericPoolManager) TapService(svcHost string) error {
err := gpm.fsCache.TouchByAddress(svcHost)
if err != nil {
return err
}
return nil
}
// IsValid checks if pod is not deleted and that it has the address passed as the argument. Also checks that all the
// containers in it are reporting a ready status for the healthCheck.
func (gpm *GenericPoolManager) IsValid(fsvc *fscache.FuncSvc) bool {
for _, obj := range fsvc.KubernetesObjects {
if obj.Kind == "pod" {
pod, err := gpm.kubernetesClient.CoreV1().Pods(obj.Namespace).Get(obj.Name, metav1.GetOptions{})
if err == nil && utils.IsReadyPod(pod) {
// Normally, the address format is http://[pod-ip]:[port], however, if the
// Istio is enabled the address format changes to http://[svc-name]:[port].
// So if the Istio is enabled and pod is in ready state, we return true directly;
// Otherwise, we need to ensure that the address contains pod ip.
if gpm.enableIstio ||
(!gpm.enableIstio && strings.Contains(fsvc.Address, pod.Status.PodIP)) {
gpm.logger.Debug("valid address", zap.String("address", fsvc.Address))
return true
}
}
}
}
return false
}
func (gpm *GenericPoolManager) RefreshFuncPods(logger *zap.Logger, f fv1.Function) error {
env, err := gpm.fissionClient.Environments(f.Spec.Environment.Namespace).Get(f.Spec.Environment.Name)
@@ -174,6 +243,140 @@ func (gpm *GenericPoolManager) RefreshFuncPods(logger *zap.Logger, f fv1.Functio
return nil
}
func (gpm *GenericPoolManager) AdoptOrphanResources() {
envs, err := gpm.fissionClient.Environments(metav1.NamespaceAll).List(metav1.ListOptions{})
if err != nil {
gpm.logger.Error("error getting environment list", zap.Error(err))
return
}
envMap := make(map[string]fv1.Environment, len(envs.Items))
envWG := &sync.WaitGroup{}
for i := range envs.Items {
env := envs.Items[i]
if gpm.getEnvPoolsize(&env) > 0 {
envWG.Add(1)
go func() {
defer envWG.Done()
_, err := gpm.getPool(&env)
if err != nil {
gpm.logger.Error("adopt pool failed", zap.Error(err))
}
}()
}
// create environment map for later use
key := fmt.Sprintf("%v/%v", env.Metadata.Namespace, env.Metadata.Name)
envMap[key] = env
}
l := map[string]string{
types.EXECUTOR_TYPE: string(fv1.ExecutorTypePoolmgr),
}
podList, err := gpm.kubernetesClient.CoreV1().Pods(metav1.NamespaceAll).List(metav1.ListOptions{
LabelSelector: labels.Set(l).AsSelector().String(),
})
if err != nil {
gpm.logger.Error("error getting pod list", zap.Error(err))
return
}
podWG := &sync.WaitGroup{}
for i := range podList.Items {
pod := &podList.Items[i]
if !utils.IsReadyPod(pod) {
continue
}
podWG.Add(1)
go func() {
defer podWG.Done()
// avoid too many requests arrive Kubernetes API server at the same time.
time.Sleep(time.Duration(rand.Intn(30)) * time.Millisecond)
patch := fmt.Sprintf(`{"metadata":{"annotations":{"%v":"%v"}}}`, types.EXECUTOR_INSTANCEID_LABEL, gpm.instanceId)
pod, err = gpm.kubernetesClient.CoreV1().Pods(pod.Namespace).Patch(pod.Name, k8sTypes.StrategicMergePatchType, []byte(patch))
if err != nil {
// just log the error since it won't affect the function serving
gpm.logger.Warn("error patching executor instance ID of pod", zap.Error(err),
zap.String("pod", pod.Name), zap.String("ns", pod.Namespace))
return
}
// for unspecialized pod, we only update its annotations
if pod.Labels["managed"] == "true" {
return
}
fnName, ok1 := pod.Labels[types.FUNCTION_NAME]
fnNS, ok2 := pod.Labels[types.FUNCTION_NAMESPACE]
fnUID, ok3 := pod.Labels[types.FUNCTION_UID]
fnRV, ok4 := pod.Annotations[types.FUNCTION_RESOURCE_VERSION]
envName, ok5 := pod.Labels[types.ENVIRONMENT_NAME]
envNS, ok6 := pod.Labels[types.ENVIRONMENT_NAMESPACE]
svcHost, ok7 := pod.Annotations[types.ANNOTATION_SVC_HOST]
env, ok8 := envMap[fmt.Sprintf("%v/%v", envNS, envName)]
if !(ok1 && ok2 && ok3 && ok4 && ok5 && ok6 && ok7 && ok8) {
gpm.logger.Warn("failed to adopt pod for function due to lack necessary information",
zap.String("pod", pod.Name), zap.Any("labels", pod.Labels), zap.Any("annotations", pod.Annotations),
zap.String("env", env.Metadata.Name))
return
}
fsvc := fscache.FuncSvc{
Name: pod.Name,
Function: &metav1.ObjectMeta{
Name: fnName,
Namespace: fnNS,
UID: k8sTypes.UID(fnUID),
ResourceVersion: fnRV,
},
Environment: &env,
Address: svcHost,
KubernetesObjects: []apiv1.ObjectReference{
{
Kind: "pod",
Name: pod.Name,
APIVersion: pod.TypeMeta.APIVersion,
Namespace: pod.ObjectMeta.Namespace,
ResourceVersion: pod.ObjectMeta.ResourceVersion,
UID: pod.ObjectMeta.UID,
},
},
Executor: fv1.ExecutorTypePoolmgr,
Ctime: time.Now(),
Atime: time.Now(),
}
// If fsvc already exists we just skip the duplicate one. And let reaper to recycle the duplicate pod.
// This is for the case that there are multiple function pods for the same function due to unknown reason.
_, err := gpm.fsCache.GetByFunction(fsvc.Function)
if err == nil {
return
}
_, err = gpm.fsCache.Add(fsvc)
if err != nil {
gpm.logger.Warn("failed to adopt pod for function", zap.Error(err), zap.String("pod", pod.Name))
return
}
gpm.logger.Info("adopt function pod",
zap.String("pod", pod.Name), zap.Any("labels", pod.Labels), zap.Any("annotations", pod.Annotations))
}()
}
envWG.Wait()
podWG.Wait()
}
func (gpm *GenericPoolManager) service() {
for {
req := <-gpm.requestChannel
@@ -246,25 +449,6 @@ func (gpm *GenericPoolManager) cleanupPools(envs []fv1.Environment) {
}
}
func (gpm *GenericPoolManager) GetFuncSvc(ctx context.Context, fn *fv1.Function) (*fscache.FuncSvc, error) {
// from Func -> get Env
gpm.logger.Debug("getting environment for function", zap.String("function", fn.Metadata.Name))
env, err := gpm.getFunctionEnv(fn)
if err != nil {
return nil, err
}
pool, err := gpm.getPool(env)
if err != nil {
return nil, err
}
// from GenericPool -> get one function container
// (this also adds to the cache)
gpm.logger.Debug("getting function service from pool", zap.String("function", fn.Metadata.Name))
return pool.getFuncSvc(ctx, fn)
}
func (gpm *GenericPoolManager) getFunctionEnv(fn *fv1.Function) (*fv1.Environment, error) {
var env *fv1.Environment
@@ -297,29 +481,38 @@ func (gpm *GenericPoolManager) eagerPoolCreator() {
if err != nil {
if utils.IsNetworkError(err) {
gpm.logger.Error("encountered network error, retrying", zap.Error(err))
time.Sleep(5 * time.Second)
continue
} else {
gpm.logger.Error("failed to get environment list", zap.Error(err))
}
gpm.logger.Error("failed to get environment list", zap.Error(err))
time.Sleep(5 * time.Second)
continue
}
// Create pools for all envs. TODO: we should make this a bit less eager, only
// creating pools for envs that are actually used by functions. Also we might want
// to keep these eagerly created pools smaller than the ones created when there are
// actual function calls.
wg := &sync.WaitGroup{}
for i := range envs.Items {
env := envs.Items[i]
// Create pool only if poolsize greater than zero
if gpm.getEnvPoolsize(&env) > 0 {
_, err := gpm.getPool(&envs.Items[i])
if err != nil {
gpm.logger.Error("eager-create pool failed", zap.Error(err))
}
wg.Add(1)
go func() {
defer wg.Done()
_, err := gpm.getPool(&env)
if err != nil {
gpm.logger.Error("eager-create pool failed", zap.Error(err))
}
}()
}
}
// Clean up pools whose env was deleted
gpm.cleanupPools(envs.Items)
wg.Wait()
time.Sleep(pollSleep)
}
}
@@ -334,28 +527,6 @@ func (gpm *GenericPoolManager) getEnvPoolsize(env *fv1.Environment) int32 {
return poolsize
}
// IsValid checks if pod is not deleted and that it has the address passed as the argument. Also checks that all the
// containers in it are reporting a ready status for the healthCheck.
func (gpm *GenericPoolManager) IsValid(fsvc *fscache.FuncSvc) bool {
for _, obj := range fsvc.KubernetesObjects {
if obj.Kind == "pod" {
pod, err := gpm.kubernetesClient.CoreV1().Pods(obj.Namespace).Get(obj.Name, metav1.GetOptions{})
if err == nil && utils.IsReadyPod(pod) {
// Normally, the address format is http://[pod-ip]:[port], however, if the
// Istio is enabled the address format changes to http://[svc-name]:[port].
// So if the Istio is enabled and pod is in ready state, we return true directly;
// Otherwise, we need to ensure that the address contains pod ip.
if gpm.enableIstio ||
(!gpm.enableIstio && strings.Contains(fsvc.Address, pod.Status.PodIP)) {
gpm.logger.Debug("valid address", zap.String("address", fsvc.Address))
return true
}
}
}
}
return false
}
// idleObjectReaper reaps objects after certain idle time
func (gpm *GenericPoolManager) idleObjectReaper() {
@@ -379,8 +550,10 @@ func (gpm *GenericPoolManager) idleObjectReaper() {
continue
}
for _, fsvc := range funcSvcs {
if fsvc.Executor != fscache.POOLMGR {
for i := range funcSvcs {
fsvc := funcSvcs[i]
if fsvc.Executor != fv1.ExecutorTypePoolmgr {
continue
}
@@ -396,20 +569,23 @@ func (gpm *GenericPoolManager) idleObjectReaper() {
continue
}
deleted, err := gpm.fsCache.DeleteOld(fsvc, gpm.idlePodReapTime)
if err != nil {
gpm.logger.Error("error deleting Kubernetes objects for function service",
zap.Error(err),
zap.Any("service", fsvc))
}
if !deleted {
continue
}
for _, kubeobj := range fsvc.KubernetesObjects {
reaper.CleanupKubeObject(gpm.logger, gpm.kubernetesClient, &kubeobj)
}
go func() {
deleted, err := gpm.fsCache.DeleteOld(fsvc, gpm.idlePodReapTime)
if err != nil {
gpm.logger.Error("error deleting Kubernetes objects for function service",
zap.Error(err),
zap.Any("service", fsvc))
}
if deleted {
for i := range fsvc.KubernetesObjects {
gpm.logger.Debug("release idle function resources",
zap.String("function", fsvc.Name), zap.String("address", fsvc.Address),
zap.String("executor", string(fsvc.Executor)))
reaper.CleanupKubeObject(gpm.logger, gpm.kubernetesClient, &fsvc.KubernetesObjects[i])
time.Sleep(50 * time.Millisecond)
}
}
}()
}
}
}
+3 -7
View File
@@ -33,7 +33,8 @@ import (
)
type fscRequestType int
type executorType int
//type executorType int
const (
TOUCH fscRequestType = iota
@@ -41,11 +42,6 @@ const (
LOG
)
const (
POOLMGR executorType = iota
NEWDEPLOY
)
type (
FuncSvc struct {
Name string // Name of object
@@ -53,7 +49,7 @@ type (
Environment *fv1.Environment // function's environment
Address string // Host:Port or IP:Port that the function's service can be reached at.
KubernetesObjects []apiv1.ObjectReference // Kubernetes Objects (within the function namespace)
Executor executorType
Executor fv1.ExecutorType
Ctime time.Time
Atime time.Time
+22 -25
View File
@@ -26,6 +26,7 @@ import (
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
"github.com/fission/fission/pkg/crd"
"github.com/fission/fission/pkg/types"
)
@@ -37,16 +38,20 @@ var (
// CleanupOldExecutorObjects cleans up resources created by old executor instances
func CleanupOldExecutorObjects(logger *zap.Logger, kubernetesClient *kubernetes.Clientset, instanceId string) {
go func() {
err := cleanup(logger, kubernetesClient, instanceId)
if err != nil {
// TODO retry reaper; logged and ignored for now
logger.Error("Failed to cleanup old executor objects", zap.Error(err))
}
}()
err := cleanup(logger, kubernetesClient, instanceId)
if err != nil {
// TODO retry reaper; logged and ignored for now
logger.Error("Failed to cleanup old executor objects", zap.Error(err))
}
}
func cleanup(logger *zap.Logger, client *kubernetes.Clientset, instanceId string) error {
// Pods might still be running user functions, so we give them
// a few minutes before terminating them. This time is the
// maximum function runtime, plus the time a router might
// still route to an old instance, i.e. router cache expiry
// time.
time.Sleep(6 * time.Minute)
err := cleanupServices(logger, client, instanceId)
if err != nil {
@@ -66,13 +71,6 @@ func cleanup(logger *zap.Logger, client *kubernetes.Clientset, instanceId string
return err
}
// Pods might still be running user functions, so we give them
// a few minutes before terminating them. This time is the
// maximum function runtime, plus the time a router might
// still route to an old instance, i.e. router cache expiry
// time.
time.Sleep(6 * time.Minute)
err = cleanupPods(logger, client, instanceId)
if err != nil {
return err
@@ -120,7 +118,7 @@ func cleanupDeployments(logger *zap.Logger, client *kubernetes.Clientset, instan
return err
}
for _, dep := range deploymentList.Items {
id, ok := dep.ObjectMeta.Labels[types.EXECUTOR_INSTANCEID_LABEL]
id, ok := dep.ObjectMeta.Annotations[types.EXECUTOR_INSTANCEID_LABEL]
if ok && id != instanceId {
logger.Debug("cleaning up deployment", zap.String("deployment", dep.ObjectMeta.Name))
err := client.AppsV1().Deployments(dep.ObjectMeta.Namespace).Delete(dep.ObjectMeta.Name, &delOpt)
@@ -133,7 +131,7 @@ func cleanupDeployments(logger *zap.Logger, client *kubernetes.Clientset, instan
// ignore err
}
// Backward compatibility with older label name
pid, pok := dep.ObjectMeta.Labels[types.POOLMGR_INSTANCEID_LABEL]
pid, pok := dep.ObjectMeta.Annotations[types.POOLMGR_INSTANCEID_LABEL]
if pok && pid != instanceId {
logger.Debug("cleaning up deployment", zap.String("deployment", dep.ObjectMeta.Name))
err := client.AppsV1().Deployments(dep.ObjectMeta.Namespace).Delete(dep.ObjectMeta.Name, &delOpt)
@@ -155,7 +153,7 @@ func cleanupPods(logger *zap.Logger, client *kubernetes.Clientset, instanceId st
return err
}
for _, pod := range podList.Items {
id, ok := pod.ObjectMeta.Labels[types.EXECUTOR_INSTANCEID_LABEL]
id, ok := pod.ObjectMeta.Annotations[types.EXECUTOR_INSTANCEID_LABEL]
if ok && id != instanceId {
logger.Debug("cleaning up pod", zap.String("pod", pod.ObjectMeta.Name))
err := client.CoreV1().Pods(pod.ObjectMeta.Namespace).Delete(pod.ObjectMeta.Name, nil)
@@ -168,7 +166,7 @@ func cleanupPods(logger *zap.Logger, client *kubernetes.Clientset, instanceId st
// ignore err
}
// Backward compatibility with older label name
pid, pok := pod.ObjectMeta.Labels[types.POOLMGR_INSTANCEID_LABEL]
pid, pok := pod.ObjectMeta.Annotations[types.POOLMGR_INSTANCEID_LABEL]
if pok && pid != instanceId {
logger.Debug("cleaning up pod", zap.String("pod", pod.ObjectMeta.Name))
err := client.CoreV1().Pods(pod.ObjectMeta.Namespace).Delete(pod.ObjectMeta.Name, nil)
@@ -189,7 +187,7 @@ func cleanupServices(logger *zap.Logger, client *kubernetes.Clientset, instanceI
return err
}
for _, svc := range svcList.Items {
id, ok := svc.ObjectMeta.Labels[types.EXECUTOR_INSTANCEID_LABEL]
id, ok := svc.ObjectMeta.Annotations[types.EXECUTOR_INSTANCEID_LABEL]
if ok && id != instanceId {
logger.Debug("cleaning up service", zap.String("service", svc.ObjectMeta.Name))
err := client.CoreV1().Services(svc.ObjectMeta.Namespace).Delete(svc.ObjectMeta.Name, nil)
@@ -212,7 +210,7 @@ func cleanupHpa(logger *zap.Logger, client *kubernetes.Clientset, instanceId str
}
for _, hpa := range hpaList.Items {
id, ok := hpa.ObjectMeta.Labels[types.EXECUTOR_INSTANCEID_LABEL]
id, ok := hpa.ObjectMeta.Annotations[types.EXECUTOR_INSTANCEID_LABEL]
if ok && id != instanceId {
logger.Debug("cleaning up HPA", zap.String("hpa", hpa.ObjectMeta.Name))
err := client.AutoscalingV1().HorizontalPodAutoscalers(hpa.ObjectMeta.Namespace).Delete(hpa.ObjectMeta.Name, nil)
@@ -224,7 +222,6 @@ func cleanupHpa(logger *zap.Logger, client *kubernetes.Clientset, instanceId str
}
// ignore err
}
}
return nil
@@ -234,6 +231,9 @@ func cleanupHpa(logger *zap.Logger, client *kubernetes.Clientset, instanceId str
// deletes the rolebindings completely if there are no Service Accounts in a rolebinding object.
func CleanupRoleBindings(logger *zap.Logger, client *kubernetes.Clientset, fissionClient *crd.FissionClient, functionNs, envBuilderNs string, cleanupRoleBindingInterval time.Duration) {
for {
// some sleep before the next reaper iteration
time.Sleep(cleanupRoleBindingInterval)
logger.Debug("starting cleanupRoleBindings cycle")
// get all rolebindings ( just to be efficient, one call to kubernetes )
rbList, err := client.RbacV1beta1().RoleBindings(meta_v1.NamespaceAll).List(meta_v1.ListOptions{})
@@ -294,7 +294,7 @@ func CleanupRoleBindings(logger *zap.Logger, client *kubernetes.Clientset, fissi
break
}
if fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType == types.ExecutorTypeNewdeploy {
if fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType == fv1.ExecutorTypeNewdeploy {
ndmFunc = true
break
}
@@ -362,8 +362,5 @@ func CleanupRoleBindings(logger *zap.Logger, client *kubernetes.Clientset, fissi
}
}
}
// some sleep before the next reaper iteration
time.Sleep(cleanupRoleBindingInterval)
}
}
+33
View File
@@ -0,0 +1,33 @@
/*
Copyright 2019 The Fission Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package util
import (
apiv1 "k8s.io/api/core/v1"
)
// ApplyImagePullSecret applies image pull secret to the give pod spec.
// It's intentional not to check the existence of secret here.
// First, Kubernetes will set Pod status to "ImagePullBackOff" once
// kubelet failed to pull image so that users will know what's happening.
// Second, Fission no longer need to handle "secret not found" error
// when creating the environment deployment since kubelet will retry to
// pull image until successes.
func ApplyImagePullSecret(secret string, podspec apiv1.PodSpec) *apiv1.PodSpec {
podspec.ImagePullSecrets = []apiv1.LocalObjectReference{{Name: secret}}
return &podspec
}
+11 -3
View File
@@ -9,9 +9,9 @@ import (
"strings"
"time"
"go.uber.org/zap"
"github.com/pkg/errors"
"go.opencensus.io/plugin/ochttp"
"go.uber.org/zap"
"golang.org/x/net/context/ctxhttp"
ferror "github.com/fission/fission/pkg/error"
@@ -91,12 +91,20 @@ func sendRequest(logger *zap.Logger, ctx context.Context, httpClient *http.Clien
if err != nil {
logger.Error("error reading response body", zap.Error(err))
}
resp.Body.Close()
defer resp.Body.Close()
return body, err
}
err = ferror.MakeErrorFromHTTP(resp)
}
// skip retry and return directly due to context deadline exceeded
if err == context.DeadlineExceeded {
msg := "error specializing function pod, either increase the specialization timeout for function or check function pod log would help."
err = errors.Wrap(err, msg)
logger.Error(msg, zap.Error(err), zap.String("url", url))
return nil, err
}
if i < maxRetries-1 {
time.Sleep(50 * time.Duration(2*i) * time.Millisecond)
logger.Error("error specializing/fetching/uploading package, retrying", zap.Error(err), zap.String("url", url))
+3 -37
View File
@@ -21,7 +21,6 @@ import (
"context"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"net/http"
"os"
@@ -33,7 +32,6 @@ import (
uuid "github.com/satori/go.uuid"
"go.opencensus.io/plugin/ochttp"
"go.uber.org/zap"
"golang.org/x/net/context/ctxhttp"
k8serr "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
@@ -96,38 +94,6 @@ func MakeFetcher(logger *zap.Logger, sharedVolumePath string, sharedSecretPath s
}, nil
}
func downloadUrl(ctx context.Context, httpClient *http.Client, url string, localPath string) error {
resp, err := ctxhttp.Get(ctx, httpClient, url)
if err != nil {
return err
}
defer resp.Body.Close()
w, err := os.Create(localPath)
if err != nil {
return err
}
defer w.Close()
_, err = io.Copy(w, resp.Body)
if err != nil {
return err
}
// flushing write buffer to file
err = w.Sync()
if err != nil {
return err
}
err = os.Chmod(localPath, 0600)
if err != nil {
return err
}
return nil
}
func verifyChecksum(fileChecksum, checksum *fv1.Checksum) error {
if checksum.Type != fv1.ChecksumTypeSHA256 {
return ferror.MakeError(ferror.ErrorInvalidArgument, "Unsupported checksum type")
@@ -232,7 +198,7 @@ func (fetcher *Fetcher) SpecializeHandler(w http.ResponseWriter, r *http.Request
err = fetcher.SpecializePod(r.Context(), req.FetchReq, req.LoadReq)
if err != nil {
fetcher.logger.Error("error specializing pod", zap.Error(err))
w.WriteHeader(http.StatusInternalServerError)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
@@ -263,7 +229,7 @@ func (fetcher *Fetcher) Fetch(ctx context.Context, pkg *fv1.Package, req types.F
if req.FetchType == types.FETCH_URL {
// fetch the file and save it to the tmp path
err := downloadUrl(ctx, fetcher.httpClient, req.Url, tmpPath)
err := utils.DownloadUrl(ctx, fetcher.httpClient, req.Url, tmpPath)
if err != nil {
e := "failed to download url"
fetcher.logger.Error(e, zap.Error(err), zap.String("url", req.Url))
@@ -302,7 +268,7 @@ func (fetcher *Fetcher) Fetch(ctx context.Context, pkg *fv1.Package, req types.F
}
} else {
// download and verify
err := downloadUrl(ctx, fetcher.httpClient, archive.URL, tmpPath)
err := utils.DownloadUrl(ctx, fetcher.httpClient, archive.URL, tmpPath)
if err != nil {
e := "failed to download url"
fetcher.logger.Error(e, zap.Error(err), zap.String("url", req.Url))
@@ -53,6 +53,14 @@ func Wrapper(action cmd.CommandAction) func(*cobra.Command, []string) error {
func SetFlags(cmd *cobra.Command, flagSet flag.FlagSet) {
aliases := make(map[string]string)
// set global flags
for _, f := range flagSet.Global {
globalFlags(cmd, f)
for _, alias := range f.Aliases {
aliases[alias] = f.Name
}
}
// set required flags
for _, f := range flagSet.Required {
requiredFlags(cmd, f)
@@ -84,7 +92,7 @@ func SetFlags(cmd *cobra.Command, flagSet flag.FlagSet) {
func optionalFlags(cmd *cobra.Command, flags ...flag.Flag) {
for _, f := range flags {
toCobraFlag(cmd, f)
toCobraFlag(cmd, f, false)
if f.Deprecated {
usage := fmt.Sprintf("Use --%v instead. The flag still works for now and will be removed in future", f.Substitute)
cmd.Flags().MarkDeprecated(f.Name, usage)
@@ -96,12 +104,18 @@ func optionalFlags(cmd *cobra.Command, flags ...flag.Flag) {
func requiredFlags(cmd *cobra.Command, flags ...flag.Flag) {
for _, f := range flags {
toCobraFlag(cmd, f)
toCobraFlag(cmd, f, false)
cmd.MarkFlagRequired(f.Name)
}
}
func toCobraFlag(cmd *cobra.Command, f flag.Flag) {
func globalFlags(cmd *cobra.Command, flags ...flag.Flag) {
for _, f := range flags {
toCobraFlag(cmd, f, true)
}
}
func toCobraFlag(cmd *cobra.Command, f flag.Flag, global bool) {
// Workaround to pass aliases to templater for generating flag aliases.
if len(f.Aliases) > 0 {
var aliases []string
@@ -119,67 +133,72 @@ func toCobraFlag(cmd *cobra.Command, f flag.Flag) {
helptemplate.AliasSeparator, f.Usage)
}
flagset := cmd.Flags()
if global {
flagset = cmd.PersistentFlags()
}
switch f.Type {
case flag.Bool:
val, ok := f.DefaultValue.(bool)
if !ok {
val = false
}
cmd.Flags().BoolP(f.Name, f.Short, val, f.Usage)
flagset.BoolP(f.Name, f.Short, val, f.Usage)
case flag.String:
val, ok := f.DefaultValue.(string)
if !ok {
val = ""
}
cmd.Flags().StringP(f.Name, f.Short, val, f.Usage)
flagset.StringP(f.Name, f.Short, val, f.Usage)
case flag.StringSlice:
val, ok := f.DefaultValue.([]string)
if !ok {
val = []string{}
}
cmd.Flags().StringArrayP(f.Name, f.Short, val, f.Usage)
flagset.StringArrayP(f.Name, f.Short, val, f.Usage)
case flag.Int:
val, ok := f.DefaultValue.(int)
if !ok {
val = 0
}
cmd.Flags().IntP(f.Name, f.Short, val, f.Usage)
flagset.IntP(f.Name, f.Short, val, f.Usage)
case flag.IntSlice:
val, ok := f.DefaultValue.([]int)
if !ok {
val = []int{}
}
cmd.Flags().IntSliceP(f.Name, f.Short, val, f.Usage)
flagset.IntSliceP(f.Name, f.Short, val, f.Usage)
case flag.Int64:
val, ok := f.DefaultValue.(int64)
if !ok {
val = 0
}
cmd.Flags().Int64P(f.Name, f.Short, val, f.Usage)
flagset.Int64P(f.Name, f.Short, val, f.Usage)
case flag.Int64Slice:
val, ok := f.DefaultValue.([]int64)
if !ok {
val = []int64{}
}
cmd.Flags().Int64SliceP(f.Name, f.Short, val, f.Usage)
flagset.Int64SliceP(f.Name, f.Short, val, f.Usage)
case flag.Float32:
val, ok := f.DefaultValue.(float32)
if !ok {
val = 0
}
cmd.Flags().Float32P(f.Name, f.Short, val, f.Usage)
flagset.Float32P(f.Name, f.Short, val, f.Usage)
case flag.Float64:
val, ok := f.DefaultValue.(float64)
if !ok {
val = 0
}
cmd.Flags().Float64P(f.Name, f.Short, val, f.Usage)
flagset.Float64P(f.Name, f.Short, val, f.Usage)
case flag.Duration:
val, ok := f.DefaultValue.(time.Duration)
if !ok {
val = 0
}
cmd.Flags().DurationP(f.Name, f.Short, val, f.Usage)
flagset.DurationP(f.Name, f.Short, val, f.Usage)
}
}
@@ -50,7 +50,7 @@ const (
{{end}}`
// SectionFlags is the help template section that displays the command's flags.
SectionFlags = `{{ if or $visibleFlags.HasFlags $explicitlyExposedFlags.HasFlags}}Options:
SectionFlags = `{{ if or $visibleFlags.HasFlags $explicitlyExposedFlags.HasFlags}}Options
{{ if $visibleFlags.HasFlags}}{{trimRight (flagsUsages $visibleFlags)}}{{end}}{{ if $explicitlyExposedFlags.HasFlags}}{{ if $visibleFlags.HasFlags}}
{{end}}{{trimRight (flagsUsages $explicitlyExposedFlags)}}{{end}}
+4 -4
View File
@@ -33,8 +33,8 @@ func Commands() *cobra.Command {
Required: []flag.Flag{flag.EnvName, flag.EnvImage},
Optional: []flag.Flag{flag.EnvPoolsize, flag.EnvBuilderImage, flag.EnvBuildCmd,
flag.RunTimeMinCPU, flag.RunTimeMaxCPU, flag.RunTimeMinMemory, flag.RunTimeMaxMemory,
flag.EnvTerminationGracePeriod, flag.EnvVersion, flag.EnvExternalNetwork, flag.EnvKeepArchive,
flag.NamespaceEnvironment, flag.SpecSave},
flag.EnvTerminationGracePeriod, flag.EnvVersion, flag.EnvImagePullSecret,
flag.EnvExternalNetwork, flag.EnvKeepArchive, flag.NamespaceEnvironment, flag.SpecSave},
})
getCmd := &cobra.Command{
@@ -55,8 +55,8 @@ func Commands() *cobra.Command {
wrapper.SetFlags(updateCmd, flag.FlagSet{
Required: []flag.Flag{flag.EnvName},
Optional: []flag.Flag{flag.EnvImage, flag.EnvPoolsize,
flag.EnvBuilderImage, flag.EnvBuildCmd, flag.EnvExternalNetwork,
flag.EnvTerminationGracePeriod, flag.EnvKeepArchive, flag.NamespaceEnvironment},
flag.EnvBuilderImage, flag.EnvBuildCmd, flag.EnvImagePullSecret, flag.EnvTerminationGracePeriod,
flag.EnvKeepArchive, flag.NamespaceEnvironment, flag.EnvExternalNetwork},
})
deleteCmd := &cobra.Command{
@@ -112,6 +112,7 @@ func createEnvironmentFromCmd(input cli.Input) (*fv1.Environment, error) {
envExternalNetwork := input.Bool(flagkey.EnvExternalNetwork)
keepArchive := input.Bool(flagkey.EnvKeeparchive)
envGracePeriod := input.Int64(flagkey.EnvGracePeriod)
pullSecret := input.String(flagkey.EnvImagePullSecret)
envVersion := input.Int(flagkey.EnvVersion)
// Environment API interface version is not specified and
@@ -169,6 +170,7 @@ func createEnvironmentFromCmd(input cli.Input) (*fv1.Environment, error) {
AllowAccessToExternalNetwork: envExternalNetwork,
TerminationGracePeriod: envGracePeriod,
KeepArchive: keepArchive,
ImagePullSecret: pullSecret,
},
}
@@ -123,6 +123,10 @@ func updateExistingEnvironmentWithCmd(env *fv1.Environment, input cli.Input) (*f
env.Spec.KeepArchive = input.Bool(flagkey.EnvKeeparchive)
}
if input.IsSet(flagkey.EnvImagePullSecret) {
env.Spec.ImagePullSecret = input.String(flagkey.EnvImagePullSecret)
}
env.Spec.AllowAccessToExternalNetwork = envExternalNetwork
// TODO: allow to update resource.
+6 -2
View File
@@ -36,13 +36,16 @@ func Commands() *cobra.Command {
flag.FnExecutorType, flag.FnCfgMap, flag.FnSecret,
flag.FnSpecializationTimeout, flag.FnExecutionTimeout,
// TODO retired pkg related flag from function cmd
// TODO retired pkg & trigger related flags from function cmd
flag.PkgCode, flag.PkgSrcArchive, flag.PkgDeployArchive,
flag.PkgSrcChecksum, flag.PkgDeployChecksum, flag.PkgInsecure,
flag.FnBuildCmd,
flag.HtUrl, flag.HtMethod,
// flag for newdeploy to use.
flag.RunTimeMinCPU, flag.RunTimeMaxCPU, flag.RunTimeMinMemory,
flag.RunTimeMaxMemory, flag.FnBuildCmd, flag.ReplicasMin,
flag.RunTimeMaxMemory, flag.ReplicasMin,
flag.ReplicasMax, flag.RunTimeTargetCPU,
flag.NamespaceFunction, flag.NamespaceEnvironment, flag.SpecSave},
@@ -84,6 +87,7 @@ func Commands() *cobra.Command {
flag.FnSpecializationTimeout, flag.FnExecutionTimeout,
flag.PkgCode, flag.PkgSrcArchive, flag.PkgDeployArchive,
flag.PkgSrcChecksum, flag.PkgDeployChecksum, flag.PkgInsecure,
flag.FnBuildCmd, flag.PkgForce,
flag.RunTimeMinCPU, flag.RunTimeMaxCPU, flag.RunTimeMinMemory,
+135 -48
View File
@@ -36,7 +36,6 @@ import (
"github.com/fission/fission/pkg/fission-cli/console"
flagkey "github.com/fission/fission/pkg/fission-cli/flag/key"
"github.com/fission/fission/pkg/fission-cli/util"
"github.com/fission/fission/pkg/types"
)
const (
@@ -363,36 +362,48 @@ func (opts *CreateSubCommand) run(input cli.Input) error {
}
func getInvokeStrategy(input cli.Input, existingInvokeStrategy *fv1.InvokeStrategy) (strategy *fv1.InvokeStrategy, err error) {
var es *fv1.ExecutionStrategy
var fnExecutor, newFnExecutor fv1.ExecutorType
if existingInvokeStrategy == nil {
es, err = getExecutionStrategy(input)
} else {
es, err = updateExecutionStrategy(input, &existingInvokeStrategy.ExecutionStrategy)
}
if err != nil {
return nil, err
}
return &fv1.InvokeStrategy{
ExecutionStrategy: *es,
StrategyType: fv1.StrategyTypeExecution,
}, nil
}
func getExecutionStrategy(input cli.Input) (strategy *fv1.ExecutionStrategy, err error) {
var fnExecutor fv1.ExecutorType
switch input.String(flagkey.FnExecutorType) {
case "":
fallthrough
case types.ExecutorTypePoolmgr:
newFnExecutor = types.ExecutorTypePoolmgr
case types.ExecutorTypeNewdeploy:
newFnExecutor = types.ExecutorTypeNewdeploy
case string(fv1.ExecutorTypePoolmgr):
fnExecutor = fv1.ExecutorTypePoolmgr
case string(fv1.ExecutorTypeNewdeploy):
fnExecutor = fv1.ExecutorTypeNewdeploy
default:
return nil, errors.New("executor type must be one of 'poolmgr' or 'newdeploy', defaults to 'poolmgr'")
return nil, errors.Errorf("executor type must be one of '%v' or '%v'", fv1.ExecutorTypePoolmgr, fv1.ExecutorTypeNewdeploy)
}
if existingInvokeStrategy != nil {
fnExecutor = existingInvokeStrategy.ExecutionStrategy.ExecutorType
specializationTimeout := fv1.DefaultSpecializationTimeOut
// override the executor type if user specified a new executor type
if input.IsSet(flagkey.FnExecutorType) {
fnExecutor = newFnExecutor
if input.IsSet(flagkey.FnSpecializationTimeout) {
specializationTimeout = input.Int(flagkey.FnSpecializationTimeout)
if specializationTimeout < fv1.DefaultSpecializationTimeOut {
return nil, errors.Errorf("%v must be greater than or equal to 120 seconds", flagkey.FnSpecializationTimeout)
}
} else {
fnExecutor = newFnExecutor
}
if input.IsSet(flagkey.FnSpecializationTimeout) && fnExecutor != types.ExecutorTypeNewdeploy {
return nil, errors.Errorf("%v flag is only applicable for newdeploy type of executor", flagkey.FnSpecializationTimeout)
}
if fnExecutor == types.ExecutorTypePoolmgr {
if fnExecutor == fv1.ExecutorTypePoolmgr {
if input.IsSet(flagkey.RuntimeTargetcpu) || input.IsSet(flagkey.ReplicasMinscale) || input.IsSet(flagkey.ReplicasMaxscale) {
return nil, errors.New("to set target CPU or min/max scale for function, please specify \"--executortype newdeploy\"")
}
@@ -400,24 +411,102 @@ func getInvokeStrategy(input cli.Input, existingInvokeStrategy *fv1.InvokeStrate
if input.IsSet(flagkey.RuntimeMincpu) || input.IsSet(flagkey.RuntimeMaxcpu) || input.IsSet(flagkey.RuntimeMinmemory) || input.IsSet(flagkey.RuntimeMaxmemory) {
console.Warn("To limit CPU/Memory for function with executor type \"poolmgr\", please specify resources limits when creating environment")
}
strategy = &fv1.InvokeStrategy{
StrategyType: fv1.StrategyTypeExecution,
ExecutionStrategy: fv1.ExecutionStrategy{
ExecutorType: types.ExecutorTypePoolmgr,
},
strategy = &fv1.ExecutionStrategy{
ExecutorType: fv1.ExecutorTypePoolmgr,
SpecializationTimeout: specializationTimeout,
}
} else {
// set default value
targetCPU := DEFAULT_TARGET_CPU_PERCENTAGE
minScale := DEFAULT_MIN_SCALE
maxScale := minScale
specializationTimeout := fv1.DefaultSpecializationTimeOut
if input.IsSet(flagkey.RuntimeTargetcpu) {
targetCPU, err = getTargetCPU(input)
if err != nil {
return nil, err
}
}
if existingInvokeStrategy != nil && existingInvokeStrategy.ExecutionStrategy.ExecutorType == types.ExecutorTypeNewdeploy {
minScale = existingInvokeStrategy.ExecutionStrategy.MinScale
maxScale = existingInvokeStrategy.ExecutionStrategy.MaxScale
targetCPU = existingInvokeStrategy.ExecutionStrategy.TargetCPUPercent
specializationTimeout = existingInvokeStrategy.ExecutionStrategy.SpecializationTimeout
minScale := DEFAULT_MIN_SCALE
if input.IsSet(flagkey.ReplicasMinscale) {
minScale = input.Int(flagkey.ReplicasMinscale)
}
maxScale := minScale
if input.IsSet(flagkey.ReplicasMaxscale) {
maxScale = input.Int(flagkey.ReplicasMaxscale)
if maxScale <= 0 {
return nil, errors.Errorf("%v must be greater than 0", flagkey.ReplicasMaxscale)
}
}
if minScale > maxScale {
return nil, fmt.Errorf("minscale (%v) can not be greater than maxscale (%v)", minScale, maxScale)
}
// Right now a simple single case strategy implementation
// This will potentially get more sophisticated once we have more strategies in place
strategy = &fv1.ExecutionStrategy{
ExecutorType: fnExecutor,
MinScale: minScale,
MaxScale: maxScale,
TargetCPUPercent: targetCPU,
SpecializationTimeout: specializationTimeout,
}
}
return strategy, nil
}
func updateExecutionStrategy(input cli.Input, existingExecutionStrategy *fv1.ExecutionStrategy) (strategy *fv1.ExecutionStrategy, err error) {
fnExecutor := existingExecutionStrategy.ExecutorType
oldExecutor := existingExecutionStrategy.ExecutorType
if input.IsSet(flagkey.FnExecutorType) {
switch input.String(flagkey.FnExecutorType) {
case "":
fallthrough
case string(fv1.ExecutorTypePoolmgr):
fnExecutor = fv1.ExecutorTypePoolmgr
case string(fv1.ExecutorTypeNewdeploy):
fnExecutor = fv1.ExecutorTypeNewdeploy
default:
return nil, errors.Errorf("executor type must be one of '%v' or '%v'", fv1.ExecutorTypePoolmgr, fv1.ExecutorTypeNewdeploy)
}
}
specializationTimeout := existingExecutionStrategy.SpecializationTimeout
if input.IsSet(flagkey.FnSpecializationTimeout) {
specializationTimeout = input.Int(flagkey.FnSpecializationTimeout)
if specializationTimeout < fv1.DefaultSpecializationTimeOut {
return nil, errors.Errorf("%v must be greater than or equal to 120 seconds", flagkey.FnSpecializationTimeout)
}
} else {
if specializationTimeout < fv1.DefaultSpecializationTimeOut {
specializationTimeout = fv1.DefaultSpecializationTimeOut
}
}
if fnExecutor == fv1.ExecutorTypePoolmgr {
if input.IsSet(flagkey.RuntimeTargetcpu) || input.IsSet(flagkey.ReplicasMinscale) || input.IsSet(flagkey.ReplicasMaxscale) {
return nil, errors.New("to set target CPU or min/max scale for function, please specify \"--executortype newdeploy\"")
}
if input.IsSet(flagkey.RuntimeMincpu) || input.IsSet(flagkey.RuntimeMaxcpu) || input.IsSet(flagkey.RuntimeMinmemory) || input.IsSet(flagkey.RuntimeMaxmemory) {
console.Warn("To limit CPU/Memory for function with executor type \"poolmgr\", please specify resources limits when creating environment")
}
strategy = &fv1.ExecutionStrategy{
ExecutorType: fv1.ExecutorTypePoolmgr,
SpecializationTimeout: specializationTimeout,
}
} else {
targetCPU := existingExecutionStrategy.TargetCPUPercent
minScale := existingExecutionStrategy.MinScale
maxScale := existingExecutionStrategy.MaxScale
if fnExecutor != oldExecutor { // from poolmanager to newdeploy
targetCPU = DEFAULT_TARGET_CPU_PERCENTAGE
minScale = DEFAULT_MIN_SCALE
maxScale = minScale
}
if input.IsSet(flagkey.RuntimeTargetcpu) {
@@ -425,6 +514,10 @@ func getInvokeStrategy(input cli.Input, existingInvokeStrategy *fv1.InvokeStrate
if err != nil {
return nil, err
}
} else {
if targetCPU <= 0 || targetCPU > 100 {
targetCPU = DEFAULT_TARGET_CPU_PERCENTAGE
}
}
if input.IsSet(flagkey.ReplicasMinscale) {
@@ -436,12 +529,9 @@ func getInvokeStrategy(input cli.Input, existingInvokeStrategy *fv1.InvokeStrate
if maxScale <= 0 {
return nil, errors.Errorf("%v must be greater than 0", flagkey.ReplicasMaxscale)
}
}
if input.IsSet(flagkey.FnSpecializationTimeout) {
specializationTimeout = input.Int(flagkey.FnSpecializationTimeout)
if specializationTimeout < fv1.DefaultSpecializationTimeOut {
return nil, errors.Errorf("%v must be greater than or equal to 120 seconds", flagkey.FnSpecializationTimeout)
} else {
if maxScale <= 0 {
maxScale = 1
}
}
@@ -451,15 +541,12 @@ func getInvokeStrategy(input cli.Input, existingInvokeStrategy *fv1.InvokeStrate
// Right now a simple single case strategy implementation
// This will potentially get more sophisticated once we have more strategies in place
strategy = &fv1.InvokeStrategy{
StrategyType: fv1.StrategyTypeExecution,
ExecutionStrategy: fv1.ExecutionStrategy{
ExecutorType: fnExecutor,
MinScale: minScale,
MaxScale: maxScale,
TargetCPUPercent: targetCPU,
SpecializationTimeout: specializationTimeout,
},
strategy = &fv1.ExecutionStrategy{
ExecutorType: fnExecutor,
MinScale: minScale,
MaxScale: maxScale,
TargetCPUPercent: targetCPU,
SpecializationTimeout: specializationTimeout,
}
}
+88 -64
View File
@@ -29,38 +29,41 @@ import (
func TestGetInvokeStrategy(t *testing.T) {
cases := []struct {
name string
testArgs map[string]interface{}
existingInvokeStrategy *fv1.InvokeStrategy
expectedResult *fv1.InvokeStrategy
expectError bool
}{
{
// case: use default executor poolmgr
name: "use default executor poolmgr",
testArgs: map[string]interface{}{},
existingInvokeStrategy: nil,
expectedResult: &fv1.InvokeStrategy{
StrategyType: fv1.StrategyTypeExecution,
ExecutionStrategy: fv1.ExecutionStrategy{
ExecutorType: fv1.ExecutorTypePoolmgr,
ExecutorType: fv1.ExecutorTypePoolmgr,
SpecializationTimeout: 120,
},
},
expectError: false,
},
{
// case: executor type set to poolmgr
testArgs: map[string]interface{}{flagkey.FnExecutorType: fv1.ExecutorTypePoolmgr},
name: "executor type set to poolmgr",
testArgs: map[string]interface{}{flagkey.FnExecutorType: string(fv1.ExecutorTypePoolmgr)},
existingInvokeStrategy: nil,
expectedResult: &fv1.InvokeStrategy{
StrategyType: fv1.StrategyTypeExecution,
ExecutionStrategy: fv1.ExecutionStrategy{
ExecutorType: fv1.ExecutorTypePoolmgr,
ExecutorType: fv1.ExecutorTypePoolmgr,
SpecializationTimeout: 120,
},
},
expectError: false,
},
{
// case: executor type set to newdeploy
testArgs: map[string]interface{}{flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy},
name: "executor type set to newdeploy",
testArgs: map[string]interface{}{flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy)},
existingInvokeStrategy: nil,
expectedResult: &fv1.InvokeStrategy{
StrategyType: fv1.StrategyTypeExecution,
@@ -75,8 +78,8 @@ func TestGetInvokeStrategy(t *testing.T) {
expectError: false,
},
{
// case: executor type change from poolmgr to newdeploy
testArgs: map[string]interface{}{flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy},
name: "executor type change from poolmgr to newdeploy",
testArgs: map[string]interface{}{flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy)},
existingInvokeStrategy: &fv1.InvokeStrategy{
StrategyType: fv1.StrategyTypeExecution,
ExecutionStrategy: fv1.ExecutionStrategy{
@@ -96,8 +99,8 @@ func TestGetInvokeStrategy(t *testing.T) {
expectError: false,
},
{
// case: executor type change from newdeploy to poolmgr
testArgs: map[string]interface{}{flagkey.FnExecutorType: fv1.ExecutorTypePoolmgr},
name: "executor type change from newdeploy to poolmgr",
testArgs: map[string]interface{}{flagkey.FnExecutorType: string(fv1.ExecutorTypePoolmgr)},
existingInvokeStrategy: &fv1.InvokeStrategy{
StrategyType: fv1.StrategyTypeExecution,
ExecutionStrategy: fv1.ExecutionStrategy{
@@ -111,15 +114,16 @@ func TestGetInvokeStrategy(t *testing.T) {
expectedResult: &fv1.InvokeStrategy{
StrategyType: fv1.StrategyTypeExecution,
ExecutionStrategy: fv1.ExecutionStrategy{
ExecutorType: fv1.ExecutorTypePoolmgr,
ExecutorType: fv1.ExecutorTypePoolmgr,
SpecializationTimeout: fv1.DefaultSpecializationTimeOut,
},
},
expectError: false,
},
{
// case: minscale < maxscale
name: "minscale < maxscale",
testArgs: map[string]interface{}{
flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy,
flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy),
flagkey.ReplicasMinscale: 2,
flagkey.ReplicasMaxscale: 3,
},
@@ -137,9 +141,9 @@ func TestGetInvokeStrategy(t *testing.T) {
expectError: false,
},
{
// case: minscale > maxscale
name: "minscale > maxscale",
testArgs: map[string]interface{}{
flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy,
flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy),
flagkey.ReplicasMinscale: 5,
flagkey.ReplicasMaxscale: 3,
},
@@ -148,19 +152,28 @@ func TestGetInvokeStrategy(t *testing.T) {
expectError: true,
},
{
// case: maxscale not specified
name: "maxscale not specified",
testArgs: map[string]interface{}{
flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy,
flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy),
flagkey.ReplicasMinscale: 5,
},
existingInvokeStrategy: nil,
expectedResult: nil,
expectError: true,
expectedResult: &fv1.InvokeStrategy{
StrategyType: fv1.StrategyTypeExecution,
ExecutionStrategy: fv1.ExecutionStrategy{
ExecutorType: fv1.ExecutorTypeNewdeploy,
MinScale: 5,
MaxScale: 5,
TargetCPUPercent: DEFAULT_TARGET_CPU_PERCENTAGE,
SpecializationTimeout: fv1.DefaultSpecializationTimeOut,
},
},
expectError: false,
},
{
// case: minscale not specified
name: "minscale not specified",
testArgs: map[string]interface{}{
flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy,
flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy),
flagkey.ReplicasMaxscale: 3,
},
existingInvokeStrategy: nil,
@@ -177,9 +190,9 @@ func TestGetInvokeStrategy(t *testing.T) {
expectError: false,
},
{
// case: maxscale set to 0
name: "maxscale set to 0",
testArgs: map[string]interface{}{
flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy,
flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy),
flagkey.ReplicasMaxscale: 0,
},
existingInvokeStrategy: nil,
@@ -187,9 +200,28 @@ func TestGetInvokeStrategy(t *testing.T) {
expectError: true,
},
{
// case: maxscale set to 9 when existing is 5
name: "update minscale with value larger than existing maxScale",
testArgs: map[string]interface{}{
flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy,
flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy),
flagkey.ReplicasMinscale: 9,
},
existingInvokeStrategy: &fv1.InvokeStrategy{
StrategyType: fv1.StrategyTypeExecution,
ExecutionStrategy: fv1.ExecutionStrategy{
ExecutorType: fv1.ExecutorTypeNewdeploy,
MinScale: 2,
MaxScale: 5,
TargetCPUPercent: DEFAULT_TARGET_CPU_PERCENTAGE,
SpecializationTimeout: fv1.DefaultSpecializationTimeOut,
},
},
expectedResult: nil,
expectError: true,
},
{
name: "maxscale set to 9 when existing is 5",
testArgs: map[string]interface{}{
flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy),
flagkey.ReplicasMaxscale: 9,
},
existingInvokeStrategy: &fv1.InvokeStrategy{
@@ -215,9 +247,9 @@ func TestGetInvokeStrategy(t *testing.T) {
expectError: false,
},
{
// case: change nothing for existing strategy
name: "change nothing for existing strategy",
testArgs: map[string]interface{}{
flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy,
flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy),
},
existingInvokeStrategy: &fv1.InvokeStrategy{
StrategyType: fv1.StrategyTypeExecution,
@@ -242,9 +274,9 @@ func TestGetInvokeStrategy(t *testing.T) {
expectError: false,
},
{
// case: set target cpu percentage
name: "set target cpu percentage",
testArgs: map[string]interface{}{
flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy,
flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy),
flagkey.RuntimeTargetcpu: 50,
},
existingInvokeStrategy: nil,
@@ -261,9 +293,9 @@ func TestGetInvokeStrategy(t *testing.T) {
expectError: false,
},
{
// case: change target cpu percentage
name: "change target cpu percentage",
testArgs: map[string]interface{}{
flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy,
flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy),
flagkey.RuntimeTargetcpu: 20,
},
existingInvokeStrategy: &fv1.InvokeStrategy{
@@ -289,9 +321,9 @@ func TestGetInvokeStrategy(t *testing.T) {
expectError: false,
},
{
// case: change specializationtimeout
name: "change specializationtimeout",
testArgs: map[string]interface{}{
flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy,
flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy),
flagkey.FnSpecializationTimeout: 200,
},
existingInvokeStrategy: &fv1.InvokeStrategy{
@@ -316,19 +348,9 @@ func TestGetInvokeStrategy(t *testing.T) {
expectError: false,
},
{
// case: specializationtimeout should not work for poolmgr
name: "specializationtimeout should not be less than 120",
testArgs: map[string]interface{}{
flagkey.FnExecutorType: fv1.ExecutorTypePoolmgr,
flagkey.FnSpecializationTimeout: 10,
},
existingInvokeStrategy: nil,
expectedResult: nil,
expectError: true,
},
{
// case: specializationtimeout should not be less than 120
testArgs: map[string]interface{}{
flagkey.FnExecutorType: fv1.ExecutorTypeNewdeploy,
flagkey.FnExecutorType: string(fv1.ExecutorTypeNewdeploy),
flagkey.FnSpecializationTimeout: 90,
},
existingInvokeStrategy: nil,
@@ -337,25 +359,27 @@ func TestGetInvokeStrategy(t *testing.T) {
},
}
for i, c := range cases {
fmt.Printf("=== Test Case %v ===\n", i)
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
flags := dummy.TestFlagSet()
flags := dummy.TestFlagSet()
for k, v := range c.testArgs {
flags.Set(k, v)
}
strategy, err := getInvokeStrategy(flags, c.existingInvokeStrategy)
if c.expectError {
assert.NotNil(t, err)
if err != nil {
fmt.Println(err)
for k, v := range c.testArgs {
flags.Set(k, v)
}
} else {
assert.Nil(t, err)
assert.NoError(t, strategy.Validate(), fmt.Sprintf("Failed at test case %v", i))
assert.Equal(t, *c.expectedResult, *strategy)
}
strategy, err := getInvokeStrategy(flags, c.existingInvokeStrategy)
if c.expectError {
assert.NotNil(t, err)
if err != nil {
fmt.Println(err)
}
} else {
assert.Nil(t, err)
if err == nil {
assert.NoError(t, strategy.Validate())
assert.Equal(t, *c.expectedResult, *strategy)
}
}
})
}
}
+5
View File
@@ -31,6 +31,7 @@ import (
"github.com/fission/fission/pkg/controller/client"
"github.com/fission/fission/pkg/fission-cli/cliwrapper/cli"
"github.com/fission/fission/pkg/fission-cli/cmd/httptrigger"
"github.com/fission/fission/pkg/fission-cli/console"
flagkey "github.com/fission/fission/pkg/fission-cli/flag/key"
"github.com/fission/fission/pkg/fission-cli/util"
)
@@ -65,6 +66,7 @@ func (opts *TestSubCommand) do(input cli.Input) error {
}
routerURL = "127.0.0.1:" + localRouterPort
} else {
console.Verbose(2, "Env FISSION_ROUTER: %v", routerURL)
routerURL = strings.TrimPrefix(routerURL, "http://")
}
@@ -77,6 +79,9 @@ func (opts *TestSubCommand) do(input cli.Input) error {
if err != nil {
return err
}
console.Verbose(2, "Function test url: %v", functionUrl.String())
queryParams := input.StringSlice(flagkey.FnTestQuery)
if len(queryParams) > 0 {
query := url.Values{}
+29 -59
View File
@@ -30,7 +30,6 @@ import (
"github.com/fission/fission/pkg/fission-cli/console"
flagkey "github.com/fission/fission/pkg/fission-cli/flag/key"
"github.com/fission/fission/pkg/fission-cli/util"
"github.com/fission/fission/pkg/types"
)
type UpdateSubCommand struct {
@@ -82,31 +81,12 @@ func (opts *UpdateSubCommand) complete(input cli.Input) error {
envNamespace = ""
}
var deployArchiveFiles []string
codeFlag := false
code := input.String(flagkey.PkgCode)
if len(code) == 0 {
deployArchiveFiles = input.StringSlice(flagkey.PkgDeployArchive)
} else {
deployArchiveFiles = append(deployArchiveFiles, input.String(flagkey.PkgCode))
codeFlag = true
}
srcArchiveFiles := input.StringSlice(flagkey.PkgSrcArchive)
pkgName := input.String(flagkey.FnPackageName)
entrypoint := input.String(flagkey.FnEntrypoint)
buildcmd := input.String(flagkey.PkgBuildCmd)
force := input.Bool(flagkey.PkgForce)
secretNames := input.StringSlice(flagkey.FnSecret)
cfgMapNames := input.StringSlice(flagkey.FnCfgMap)
specializationTimeout := input.Int(flagkey.FnSpecializationTimeout)
if len(srcArchiveFiles) > 0 && len(deployArchiveFiles) > 0 {
return errors.Errorf("need either of --%v or --%v and not both arguments", flagkey.PkgSrcArchive, flagkey.PkgDeployArchive)
}
var secrets []fv1.SecretReference
var configMaps []fv1.ConfigMapReference
@@ -187,18 +167,6 @@ func (opts *UpdateSubCommand) complete(input cli.Input) error {
}
function.Spec.InvokeStrategy = *strategy
if input.IsSet(flagkey.FnSpecializationTimeout) {
if strategy.ExecutionStrategy.ExecutorType != types.ExecutorTypeNewdeploy {
return errors.Errorf("--%v flag is only applicable for newdeploy type of executor", flagkey.FnSpecializationTimeout)
}
if specializationTimeout < fv1.DefaultSpecializationTimeOut {
return errors.Errorf("--%v must be greater than or equal to 120 seconds", flagkey.FnSpecializationTimeout)
} else {
function.Spec.InvokeStrategy.ExecutionStrategy.SpecializationTimeout = specializationTimeout
}
}
resReqs, err := util.GetResourceReqs(input, &function.Spec.Resources)
if err != nil {
return err
@@ -214,36 +182,38 @@ func (opts *UpdateSubCommand) complete(input cli.Input) error {
return errors.Wrap(err, fmt.Sprintf("read package '%v.%v'. Pkg should be present in the same ns as the function", pkgName, fnNamespace))
}
pkgMetadata := &pkg.Metadata
forceUpdate := input.Bool(flagkey.PkgForce)
if len(deployArchiveFiles) != 0 || len(srcArchiveFiles) != 0 || len(buildcmd) != 0 || len(envName) != 0 || len(envNamespace) != 0 {
fnList, err := _package.GetFunctionsByPackage(opts.client, pkg.Metadata.Name, pkg.Metadata.Namespace)
if err != nil {
return errors.Wrap(err, "error getting function list")
}
fnList, err := _package.GetFunctionsByPackage(opts.client, pkg.Metadata.Name, pkg.Metadata.Namespace)
if err != nil {
return errors.Wrap(err, "error getting function list")
}
if !force && len(fnList) > 1 {
return errors.New("package is used by multiple functions, use --force to force update")
}
if !forceUpdate && len(fnList) > 1 {
return errors.Errorf("Package is used by multiple functions, use --%v to force update", flagkey.PkgForce)
}
pkgMetadata, err = _package.UpdatePackage(opts.client, pkg, envName, envNamespace, srcArchiveFiles, deployArchiveFiles, buildcmd, false, codeFlag)
if err != nil {
return errors.Wrap(err, fmt.Sprintf("error updating package '%v'", pkgName))
}
newPkgMeta, err := _package.UpdatePackage(input, opts.client, pkg)
if err != nil {
return errors.Wrap(err, fmt.Sprintf("error updating package '%v'", pkgName))
}
fmt.Printf("package '%v' updated\n", pkgMetadata.GetName())
// update resource version of package reference of functions that shared the same package
// the package resource version of function has been changed,
// we need to update function resource version to prevent conflict.
// TODO: remove this block when deprecating pkg flags of function command.
if pkg.Metadata.ResourceVersion != newPkgMeta.ResourceVersion {
var fns []fv1.Function
// don't update the package resource version of the function we are currently
// updating to prevent update conflict.
for _, fn := range fnList {
// ignore the update for current function here, it will be updated later.
if fn.Metadata.Name != fnName {
fn.Spec.Package.PackageRef.ResourceVersion = pkgMetadata.ResourceVersion
_, err := opts.client.FunctionUpdate(&fn)
if err != nil {
return errors.Wrap(err, "error updating function")
}
if fn.Metadata.UID != function.Metadata.UID {
fns = append(fns, fn)
}
}
err = _package.UpdateFunctionPackageResourceVersion(opts.client, newPkgMeta, fns...)
if err != nil {
return errors.Wrap(err, "error updating function package reference resource version")
}
}
// TODO : One corner case where user just updates the pkg reference with fnUpdate, but internally this new pkg reference
@@ -251,9 +221,9 @@ func (opts *UpdateSubCommand) complete(input cli.Input) error {
// update function spec with new package metadata
function.Spec.Package.PackageRef = fv1.PackageRef{
Namespace: pkgMetadata.Namespace,
Name: pkgMetadata.Name,
ResourceVersion: pkgMetadata.ResourceVersion,
Namespace: newPkgMeta.Namespace,
Name: newPkgMeta.Name,
ResourceVersion: newPkgMeta.ResourceVersion,
}
if function.Spec.Environment.Name != pkg.Spec.Environment.Name {
@@ -273,6 +243,6 @@ func (opts *UpdateSubCommand) run(input cli.Input) error {
return errors.Wrap(err, "error updating function")
}
fmt.Printf("function '%v' updated\n", opts.function.Metadata.Name)
fmt.Printf("Function '%v' updated\n", opts.function.Metadata.Name)
return nil
}
+5 -3
View File
@@ -32,7 +32,8 @@ func Commands() *cobra.Command {
wrapper.SetFlags(createCmd, flag.FlagSet{
Required: []flag.Flag{flag.PkgEnvironment},
Optional: []flag.Flag{flag.PkgName, flag.PkgCode, flag.PkgSrcArchive, flag.PkgDeployArchive,
flag.PkgBuildCmd, flag.NamespacePackage, flag.NamespaceEnvironment, flag.SpecSave},
flag.PkgSrcChecksum, flag.PkgDeployChecksum, flag.PkgInsecure, flag.PkgBuildCmd,
flag.NamespacePackage, flag.NamespaceEnvironment, flag.SpecSave},
})
getSrcCmd := &cobra.Command{
@@ -62,8 +63,9 @@ func Commands() *cobra.Command {
}
wrapper.SetFlags(updateCmd, flag.FlagSet{
Required: []flag.Flag{flag.PkgName},
Optional: []flag.Flag{flag.PkgEnvironment, flag.PkgSrcArchive, flag.PkgDeployArchive,
flag.PkgBuildCmd, flag.PkgForce, flag.NamespacePackage, flag.NamespaceEnvironment},
Optional: []flag.Flag{flag.PkgEnvironment, flag.PkgCode, flag.PkgSrcArchive, flag.PkgDeployArchive,
flag.PkgSrcChecksum, flag.PkgDeployChecksum, flag.PkgInsecure, flag.PkgBuildCmd, flag.PkgForce,
flag.NamespacePackage, flag.NamespaceEnvironment},
})
deleteCmd := &cobra.Command{
+9 -4
View File
@@ -124,6 +124,10 @@ func (opts *CreateSubCommand) run(input cli.Input) error {
func CreatePackage(input cli.Input, client *client.Client, pkgName string, pkgNamespace string, envName string, envNamespace string,
srcArchiveFiles []string, deployArchiveFiles []string, buildcmd string, specDir string, specFile string, noZip bool) (*metav1.ObjectMeta, error) {
insecure := input.Bool(flagkey.PkgInsecure)
deployChecksum := input.String(flagkey.PkgDeployChecksum)
srcChecksum := input.String(flagkey.PkgSrcChecksum)
pkgSpec := fv1.PackageSpec{
Environment: fv1.EnvironmentReference{
Namespace: envNamespace,
@@ -136,9 +140,9 @@ func CreatePackage(input cli.Input, client *client.Client, pkgName string, pkgNa
if len(specFile) > 0 { // we should do this in all cases, i think
pkgStatus = fv1.BuildStatusNone
}
deployment, err := CreateArchive(client, deployArchiveFiles, noZip, specDir, specFile)
deployment, err := CreateArchive(client, deployArchiveFiles, noZip, insecure, deployChecksum, specDir, specFile)
if err != nil {
return nil, err
return nil, errors.Wrap(err, "error creating source archive")
}
pkgSpec.Deployment = *deployment
if len(pkgName) == 0 {
@@ -146,9 +150,9 @@ func CreatePackage(input cli.Input, client *client.Client, pkgName string, pkgNa
}
}
if len(srcArchiveFiles) > 0 {
source, err := CreateArchive(client, srcArchiveFiles, false, specDir, specFile)
source, err := CreateArchive(client, srcArchiveFiles, false, insecure, srcChecksum, specDir, specFile)
if err != nil {
return nil, err
return nil, errors.Wrap(err, "error creating deploy archive")
}
pkgSpec.Source = *source
pkgStatus = fv1.BuildStatusPending // set package build status to pending
@@ -164,6 +168,7 @@ func CreatePackage(input cli.Input, client *client.Client, pkgName string, pkgNa
if len(pkgName) == 0 {
pkgName = strings.ToLower(uuid.NewV4().String())
}
pkg := &fv1.Package{
Metadata: metav1.ObjectMeta{
Name: pkgName,
+71 -38
View File
@@ -19,6 +19,7 @@ package _package
import (
"context"
"fmt"
"net/http"
"os"
"path/filepath"
"strings"
@@ -33,15 +34,18 @@ import (
pkgutil "github.com/fission/fission/pkg/fission-cli/cmd/package/util"
"github.com/fission/fission/pkg/fission-cli/cmd/spec"
spectypes "github.com/fission/fission/pkg/fission-cli/cmd/spec/types"
"github.com/fission/fission/pkg/fission-cli/console"
flagkey "github.com/fission/fission/pkg/fission-cli/flag/key"
"github.com/fission/fission/pkg/fission-cli/util"
"github.com/fission/fission/pkg/utils"
uuid "github.com/satori/go.uuid"
)
// CreateArchive returns a fv1.Archive made from an archive . If specFile, then
// create an archive upload spec in the specs directory; otherwise
// upload the archive using client. noZip avoids zipping the
// includeFiles, but is ignored if there's more than one includeFile.
func CreateArchive(client *client.Client, includeFiles []string, noZip bool, specDir string, specFile string) (*fv1.Archive, error) {
func CreateArchive(client *client.Client, includeFiles []string, noZip bool, insecure bool, checksum string, specDir string, specFile string) (*fv1.Archive, error) {
// get root dir
var rootDir string
var err error
@@ -96,51 +100,80 @@ func CreateArchive(client *client.Client, includeFiles []string, noZip bool, spe
return nil, errs.ErrorOrNil()
}
if len(specFile) > 0 {
if len(fileURL) > 0 {
if len(fileURL) > 0 {
if insecure {
return &fv1.Archive{
Type: fv1.ArchiveTypeUrl,
URL: fileURL,
}, nil
} else {
// create an ArchiveUploadSpec and reference it from the archive
aus := &spectypes.ArchiveUploadSpec{
Name: archiveName("", includeFiles),
IncludeGlobs: includeFiles,
}
// check if this AUS exists in the specs; if so, don't create a new one
fr, err := spec.ReadSpecs(specDir)
if err != nil {
return nil, errors.Wrap(err, "error reading specs")
}
obj := fr.SpecExists(aus, true, true)
if obj != nil {
oldAus := obj.(*spectypes.ArchiveUploadSpec)
fmt.Printf("Re-using previously created archive %v\n", oldAus.Name)
aus.Name = oldAus.Name
} else {
// save the uploadspec
err := spec.SpecSave(*aus, specFile)
if err != nil {
return nil, errors.Wrapf(err, "write spec file %v", specFile)
}
}
// create the archive object
archive := fv1.Archive{
Type: fv1.ArchiveTypeUrl,
URL: fmt.Sprintf("%v%v", spec.ARCHIVE_URL_PREFIX, aus.Name),
}
return &archive, nil
}
var csum *fv1.Checksum
if len(checksum) > 0 {
csum = &fv1.Checksum{
Type: fv1.ChecksumTypeSHA256,
Sum: checksum,
}
} else {
console.Info(fmt.Sprintf("Downloading file to generate SHA256 checksum. To skip this step, please use --%v / --%v / --%v",
flagkey.PkgSrcChecksum, flagkey.PkgDeployChecksum, flagkey.PkgInsecure))
tmpDir, err := utils.GetTempDir()
if err != nil {
return nil, err
}
file := filepath.Join(tmpDir, uuid.NewV4().String())
err = utils.DownloadUrl(context.Background(), http.DefaultClient, fileURL, file)
if err != nil {
return nil, errors.Wrap(err, "error downloading file from the given URL")
}
csum, err = utils.GetFileChecksum(file)
if err != nil {
return nil, errors.Wrap(err, "error generating file SHA256 checksum")
}
}
return &fv1.Archive{
Type: fv1.ArchiveTypeUrl,
URL: fileURL,
Checksum: *csum,
}, nil
}
if len(fileURL) > 0 {
return &fv1.Archive{
if len(specFile) > 0 {
// create an ArchiveUploadSpec and reference it from the archive
aus := &spectypes.ArchiveUploadSpec{
Name: archiveName("", includeFiles),
IncludeGlobs: includeFiles,
}
// check if this AUS exists in the specs; if so, don't create a new one
fr, err := spec.ReadSpecs(specDir)
if err != nil {
return nil, errors.Wrap(err, "error reading specs")
}
obj := fr.SpecExists(aus, true, true)
if obj != nil {
oldAus := obj.(*spectypes.ArchiveUploadSpec)
fmt.Printf("Re-using previously created archive %v\n", oldAus.Name)
aus.Name = oldAus.Name
} else {
// save the uploadspec
err := spec.SpecSave(*aus, specFile)
if err != nil {
return nil, errors.Wrapf(err, "write spec file %v", specFile)
}
}
// create the archive object
archive := fv1.Archive{
Type: fv1.ArchiveTypeUrl,
URL: fileURL,
}, nil
URL: fmt.Sprintf("%v%v", spec.ARCHIVE_URL_PREFIX, aus.Name),
}
return &archive, nil
}
archivePath, err := makeArchiveFile("", includeFiles, noZip)
+86 -55
View File
@@ -20,6 +20,7 @@ import (
"fmt"
"time"
"github.com/hashicorp/go-multierror"
"github.com/pkg/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -31,15 +32,10 @@ import (
)
type UpdateSubCommand struct {
client *client.Client
pkgName string
pkgNamespace string
force bool
envName string
envNamespace string
srcArchiveFiles []string
deployArchiveFiles []string
buildcmd string
client *client.Client
pkgName string
pkgNamespace string
force bool
}
func Update(input cli.Input) error {
@@ -65,11 +61,6 @@ func (opts *UpdateSubCommand) complete(input cli.Input) error {
opts.pkgName = input.String(flagkey.PkgName)
opts.pkgNamespace = input.String(flagkey.NamespacePackage)
opts.force = input.Bool(flagkey.PkgForce)
opts.envName = input.String(flagkey.PkgEnvironment)
opts.envNamespace = input.String(flagkey.NamespaceEnvironment)
opts.srcArchiveFiles = input.StringSlice(flagkey.PkgSrcArchive)
opts.deployArchiveFiles = input.StringSlice(flagkey.PkgDeployArchive)
opts.buildcmd = input.String(flagkey.PkgBuildCmd)
return nil
}
@@ -82,88 +73,111 @@ func (opts *UpdateSubCommand) run(input cli.Input) error {
return errors.Wrap(err, "get package")
}
// if the new env specified is the same as the old one, no need to update package
// same is true for all update parameters, but, for now, we dont check all of them - because, its ok to
// re-write the object with same old values, we just end up getting a new resource version for the object.
if len(opts.envName) > 0 && opts.envName == pkg.Spec.Environment.Name {
opts.envName = ""
}
if opts.envNamespace == pkg.Spec.Environment.Namespace {
opts.envNamespace = ""
}
forceUpdate := input.Bool(flagkey.PkgForce)
fnList, err := GetFunctionsByPackage(opts.client, pkg.Metadata.Name, pkg.Metadata.Namespace)
if err != nil {
return errors.Wrap(err, "get function list")
return errors.Wrap(err, "error getting function list")
}
if !opts.force && len(fnList) > 1 {
return errors.New("Package is used by multiple functions, use --force to force update")
if !forceUpdate && len(fnList) > 1 {
return errors.Errorf("package is used by multiple functions, use --%v to force update", flagkey.PkgForce)
}
newPkgMeta, err := UpdatePackage(opts.client, pkg,
opts.envName, opts.envNamespace, opts.srcArchiveFiles,
opts.deployArchiveFiles, opts.buildcmd, false, false)
newPkgMeta, err := UpdatePackage(input, opts.client, pkg)
if err != nil {
return errors.Wrap(err, "update package")
return errors.Wrap(err, "error updating package")
}
// update resource version of package reference of functions that shared the same package
for _, fn := range fnList {
fn.Spec.Package.PackageRef.ResourceVersion = newPkgMeta.ResourceVersion
_, err := opts.client.FunctionUpdate(&fn)
if pkg.Metadata.ResourceVersion != newPkgMeta.ResourceVersion {
err = UpdateFunctionPackageResourceVersion(opts.client, newPkgMeta, fnList...)
if err != nil {
return errors.Wrap(err, "update function")
return errors.Wrap(err, "error updating function package reference resource version")
}
}
fmt.Printf("Package '%v' updated\n", newPkgMeta.GetName())
return nil
}
func UpdatePackage(client *client.Client, pkg *fv1.Package, envName, envNamespace string,
srcArchiveFiles []string, deployArchiveFiles []string, buildcmd string, forceRebuild bool, noZip bool) (*metav1.ObjectMeta, error) {
func UpdatePackage(input cli.Input, client *client.Client, pkg *fv1.Package) (*metav1.ObjectMeta, error) {
envName := input.String(flagkey.PkgEnvironment)
envNamespace := input.String(flagkey.NamespaceEnvironment)
srcArchiveFiles := input.StringSlice(flagkey.PkgSrcArchive)
deployArchiveFiles := input.StringSlice(flagkey.PkgDeployArchive)
buildcmd := input.String(flagkey.PkgBuildCmd)
insecure := input.Bool(flagkey.PkgInsecure)
deployChecksum := input.String(flagkey.PkgDeployChecksum)
srcChecksum := input.String(flagkey.PkgSrcChecksum)
code := input.String(flagkey.PkgCode)
needToBuild := false
noZip := false
needToRebuild := false
needToUpdate := false
if len(envName) > 0 {
if input.IsSet(flagkey.PkgCode) {
deployArchiveFiles = append(deployArchiveFiles, code)
noZip = true
needToUpdate = true
}
if input.IsSet(flagkey.PkgEnvironment) {
pkg.Spec.Environment.Name = envName
needToBuild = true
needToRebuild = true
needToUpdate = true
}
if len(envNamespace) > 0 {
if input.IsSet(flagkey.NamespaceEnvironment) {
pkg.Spec.Environment.Namespace = envNamespace
needToBuild = true
needToRebuild = true
needToUpdate = true
}
if len(buildcmd) > 0 {
if input.IsSet(flagkey.PkgBuildCmd) {
pkg.Spec.BuildCommand = buildcmd
needToBuild = true
needToRebuild = true
needToUpdate = true
}
if len(srcArchiveFiles) > 0 {
srcArchive, err := CreateArchive(client, srcArchiveFiles, false, "", "")
if input.IsSet(flagkey.PkgSrcArchive) {
srcArchive, err := CreateArchive(client, srcArchiveFiles, noZip, insecure, srcChecksum, "", "")
if err != nil {
return nil, err
return nil, errors.Wrap(err, "error creating source archive")
}
pkg.Spec.Source = *srcArchive
needToBuild = true
needToRebuild = true
needToUpdate = true
} else if input.IsSet(flagkey.PkgSrcChecksum) {
pkg.Spec.Source.Checksum = fv1.Checksum{
Type: fv1.ChecksumTypeSHA256,
Sum: srcChecksum,
}
needToUpdate = true
}
if len(deployArchiveFiles) > 0 {
deployArchive, err := CreateArchive(client, deployArchiveFiles, noZip, "", "")
if input.IsSet(flagkey.PkgDeployArchive) || input.IsSet(flagkey.PkgCode) {
deployArchive, err := CreateArchive(client, deployArchiveFiles, noZip, insecure, deployChecksum, "", "")
if err != nil {
return nil, err
return nil, errors.Wrap(err, "error creating deploy archive")
}
pkg.Spec.Deployment = *deployArchive
// Users may update the env, envNS and deploy archive at the same time,
// but without the source archive. In this case, we should set needToBuild to false
needToBuild = false
needToRebuild = false
needToUpdate = true
} else if input.IsSet(flagkey.PkgDeployChecksum) {
pkg.Spec.Deployment.Checksum = fv1.Checksum{
Type: fv1.ChecksumTypeSHA256,
Sum: srcChecksum,
}
needToUpdate = true
}
if !needToUpdate {
return &pkg.Metadata, nil
}
// Set package as pending status when needToBuild is true
if needToBuild || forceRebuild {
if needToRebuild {
// change into pending state to trigger package build
pkg.Status = fv1.PackageStatus{
BuildStatus: fv1.BuildStatusPending,
@@ -176,9 +190,26 @@ func UpdatePackage(client *client.Client, pkg *fv1.Package, envName, envNamespac
return nil, errors.Wrap(err, "update package")
}
fmt.Printf("Package '%v' updated\n", newPkgMeta.GetName())
return newPkgMeta, err
}
func UpdateFunctionPackageResourceVersion(client *client.Client, pkgMeta *metav1.ObjectMeta, fnList ...fv1.Function) error {
errs := &multierror.Error{}
// update resource version of package reference of functions that shared the same package
for _, fn := range fnList {
fn.Spec.Package.PackageRef.ResourceVersion = pkgMeta.ResourceVersion
_, err := client.FunctionUpdate(&fn)
if err != nil {
errs = multierror.Append(errs, errors.Wrapf(err, "error updating package resource version of function '%v'", fn.Metadata.Name))
}
}
return errs.ErrorOrNil()
}
func updatePackageStatus(client *client.Client, pkg *fv1.Package, status fv1.BuildStatus) (*metav1.ObjectMeta, error) {
switch status {
case fv1.BuildStatusNone, fv1.BuildStatusPending, fv1.BuildStatusRunning, fv1.BuildStatusSucceeded, fv1.CanaryConfigStatusAborted:
+16 -12
View File
@@ -24,13 +24,13 @@ import (
fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
flagkey "github.com/fission/fission/pkg/fission-cli/flag/key"
"github.com/fission/fission/pkg/types"
)
type (
FlagType = int
FlagSet struct {
Global []Flag
Required []Flag
Optional []Flag
}
@@ -93,7 +93,7 @@ var (
FnBuildCmd = Flag{Type: String, Name: flagkey.FnBuildCmd, Usage: "Package build command for builder to run with"}
FnSecret = Flag{Type: StringSlice, Name: flagkey.FnSecret, Usage: "Function access to secret, should be present in the same namespace as the function. You can provide multiple secrets using multiple --secrets flags. In the case of fn update the the secrets will be replaced by the provided list of secrets."}
FnCfgMap = Flag{Type: StringSlice, Name: flagkey.FnCfgMap, Usage: "Function access to configmap, should be present in the same namespace as the function. You can provide multiple configmaps using multiple --configmap flags. In case of fn update the configmaps will be replaced by the provided list of configmaps."}
FnExecutorType = Flag{Type: String, Name: flagkey.FnExecutorType, Usage: "Executor type for execution; one of 'poolmgr', 'newdeploy'", DefaultValue: types.ExecutorTypePoolmgr}
FnExecutorType = Flag{Type: String, Name: flagkey.FnExecutorType, Usage: "Executor type for execution; one of 'poolmgr', 'newdeploy'", DefaultValue: string(fv1.ExecutorTypePoolmgr)}
FnExecutionTimeout = Flag{Type: Int, Name: flagkey.FnExecutionTimeout, Aliases: []string{"ft"}, Usage: "Maximum time for a request to wait for the response from the function", DefaultValue: 60}
FnLogPod = Flag{Type: String, Name: flagkey.FnLogPod, Usage: "Function pod name (use the latest pod name if unspecified)"}
FnLogFollow = Flag{Type: Bool, Name: flagkey.FnLogFollow, Short: "f", Usage: "Specify if the logs should be streamed"}
@@ -141,6 +141,7 @@ var (
EnvExternalNetwork = Flag{Type: Bool, Name: flagkey.EnvExternalNetwork, Usage: "Allow pod to access external network (only works when istio feature is enabled)"}
EnvTerminationGracePeriod = Flag{Type: Int, Name: flagkey.EnvGracePeriod, Aliases: []string{"period"}, Usage: "Grace time (in seconds) for pod to perform connection draining before termination", DefaultValue: 360}
EnvVersion = Flag{Type: Int, Name: flagkey.EnvVersion, Usage: "Environment API version (1 means v1 interface)", DefaultValue: 1}
EnvImagePullSecret = Flag{Type: String, Name: flagkey.EnvImagePullSecret, Usage: "Secret for Kubernetes to pull an image from a private registry"}
KwName = Flag{Type: String, Name: flagkey.KwName, Usage: "Watch name"}
KwFnName = Flag{Type: String, Name: flagkey.KwFnName, Usage: "Function name"}
@@ -148,16 +149,19 @@ var (
KwObjType = Flag{Type: String, Name: flagkey.KwObjType, Usage: "Type of resource to watch (Pod, Service, etc.)", DefaultValue: "pod"}
KwLabels = Flag{Type: String, Name: flagkey.KwLabels, Usage: "Label selector of the form a=b,c=d"}
PkgName = Flag{Type: String, Name: flagkey.PkgName, Usage: "Package name"}
PkgForce = Flag{Type: Bool, Name: flagkey.PkgForce, Short: "f", Usage: "Force update a package even if it is used by one or more functions"}
PkgEnvironment = Flag{Type: String, Name: flagkey.PkgEnvironment, Usage: "Environment name"}
PkgBuildCmd = Flag{Type: String, Name: flagkey.PkgBuildCmd, Usage: "Build command for builder to run with"}
PkgOutput = Flag{Type: String, Name: flagkey.PkgOutput, Short: "o", Usage: "Output filename to save archive content"}
PkgStatus = Flag{Type: String, Name: flagkey.PkgStatus, Usage: `Filter packages by status`}
PkgOrphan = Flag{Type: Bool, Name: flagkey.PkgOrphan, Usage: "Orphan packages that are not referenced by any function"}
PkgCode = Flag{Type: String, Name: flagkey.PkgCode, Usage: "URL or local path for single file source code"}
PkgDeployArchive = Flag{Type: StringSlice, Name: flagkey.PkgDeployArchive, Aliases: []string{"deploy"}, Usage: "URL or local paths for binary archive"}
PkgSrcArchive = Flag{Type: StringSlice, Name: flagkey.PkgSrcArchive, Aliases: []string{"source", "src"}, Usage: "URL or local paths for source archive"}
PkgName = Flag{Type: String, Name: flagkey.PkgName, Usage: "Package name"}
PkgForce = Flag{Type: Bool, Name: flagkey.PkgForce, Short: "f", Usage: "Force update a package even if it is used by one or more functions"}
PkgEnvironment = Flag{Type: String, Name: flagkey.PkgEnvironment, Usage: "Environment name"}
PkgBuildCmd = Flag{Type: String, Name: flagkey.PkgBuildCmd, Usage: "Build command for builder to run with"}
PkgOutput = Flag{Type: String, Name: flagkey.PkgOutput, Short: "o", Usage: "Output filename to save archive content"}
PkgStatus = Flag{Type: String, Name: flagkey.PkgStatus, Usage: `Filter packages by status`}
PkgOrphan = Flag{Type: Bool, Name: flagkey.PkgOrphan, Usage: "Orphan packages that are not referenced by any function"}
PkgCode = Flag{Type: String, Name: flagkey.PkgCode, Usage: "URL or local path for single file source code"}
PkgDeployArchive = Flag{Type: StringSlice, Name: flagkey.PkgDeployArchive, Aliases: []string{"deploy"}, Usage: "URL or local paths for binary archive"}
PkgDeployChecksum = Flag{Type: String, Name: flagkey.PkgDeployChecksum, Usage: "SHA256 checksum of deploy archive when providing URL"}
PkgSrcArchive = Flag{Type: StringSlice, Name: flagkey.PkgSrcArchive, Aliases: []string{"source", "src"}, Usage: "URL or local paths for source archive"}
PkgSrcChecksum = Flag{Type: String, Name: flagkey.PkgSrcChecksum, Usage: "SHA256 checksum of source archive when providing URL"}
PkgInsecure = Flag{Type: Bool, Name: flagkey.PkgInsecure, Usage: "Skip generating SHA256 checksum for file integrity validation"}
SpecSave = Flag{Type: Bool, Name: flagkey.SpecSave, Usage: "Save to the spec directory instead of creating on cluster"}
SpecDir = Flag{Type: String, Name: flagkey.SpecDir, Usage: "Directory to store specs, defaults to ./specs"}
+14 -10
View File
@@ -96,6 +96,7 @@ const (
EnvExternalNetwork = "externalnetwork"
EnvGracePeriod = "graceperiod"
EnvVersion = "version"
EnvImagePullSecret = "imagepullsecret"
KwName = resourceName
KwFnName = "function"
@@ -103,16 +104,19 @@ const (
KwObjType = "type"
KwLabels = "labels"
PkgName = resourceName
PkgForce = force
PkgEnvironment = "env"
PkgCode = "code"
PkgSrcArchive = "sourcearchive"
PkgDeployArchive = "deployarchive"
PkgBuildCmd = "buildcmd"
PkgOutput = Output
PkgStatus = "status"
PkgOrphan = "orphan"
PkgName = resourceName
PkgForce = force
PkgEnvironment = "env"
PkgCode = "code"
PkgSrcArchive = "sourcearchive"
PkgDeployArchive = "deployarchive"
PkgSrcChecksum = "srcchecksum"
PkgDeployChecksum = "deploychecksum"
PkgInsecure = "insecure"
PkgBuildCmd = "buildcmd"
PkgOutput = Output
PkgStatus = "status"
PkgOrphan = "orphan"
SpecSave = "spec"
SpecDir = "specdir"
+181 -131
View File
@@ -32,7 +32,6 @@ import (
"github.com/pkg/errors"
"go.opencensus.io/plugin/ochttp"
"go.uber.org/zap"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
k8stypes "k8s.io/apimachinery/pkg/types"
fv1 "github.com/fission/fission/pkg/apis/fission.io/v1"
@@ -54,9 +53,9 @@ type (
logger *zap.Logger
fmap *functionServiceMap
executor *executorClient.Client
function *metav1.ObjectMeta
function *fv1.Function
httpTrigger *fv1.HTTPTrigger
functionMetadataMap map[string]*metav1.ObjectMeta
functionMap map[string]*fv1.Function
fnWeightDistributionList []FunctionWeightDistribution
tsRoundTripperParams *tsRoundTripperParams
isDebugEnv bool
@@ -86,9 +85,13 @@ type (
// A layer on top of http.DefaultTransport, with retries.
RetryingRoundTripper struct {
logger *zap.Logger
funcHandler *functionHandler
timeout int
logger *zap.Logger
funcHandler *functionHandler
funcTimeout time.Duration
closeContextFunc *context.CancelFunc
serviceUrl *url.URL
urlFromCache bool
totalRetry int
}
// To keep the request body open during retries, we create an interface with Close operation being a no-op.
@@ -145,26 +148,10 @@ func (w *fakeCloseReadCloser) RealClose() error {
// inside ServeHttp function of the reverseProxy.
// Earlier, GetServiceForFunction was called inside handler function and fission explicitly set http status code to 500
// if it returned an error.
func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
func (roundTripper *RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
// Set forwarded host header if not exists
roundTripper.addForwardedHostHeader(req)
fnMeta := roundTripper.funcHandler.function
// Metrics stuff
startTime := time.Now()
funcMetricLabels := &functionLabels{
namespace: fnMeta.Namespace,
name: fnMeta.Name,
}
httpMetricLabels := &httpLabels{
method: req.Method,
}
if roundTripper.funcHandler.httpTrigger != nil {
httpMetricLabels.host = roundTripper.funcHandler.httpTrigger.Spec.Host
httpMetricLabels.path = roundTripper.funcHandler.httpTrigger.Spec.RelativeURL
}
// set the timeout for transport context
transport := roundTripper.getDefaultTransport()
ocRoundTripper := &ochttp.Transport{Base: transport}
@@ -183,6 +170,8 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
}
}()
roundTripper.logger.Debug("request headers", zap.Any("headers", req.Header))
// The reason for request failure may vary from case to case.
// After some investigation, found most of the failure are due to
// network timeout or target function is under heavy workload. In
@@ -194,19 +183,15 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
// than the predefined threshold, reset retryCounter and remove service
// cache, then retry to get new svc record from executor again.
var retryCounter int
var serviceUrl *url.URL
var serviceUrlFromCache bool
var err error
var resp *http.Response
var fnMeta = &roundTripper.funcHandler.function.Metadata
for i := 0; i < roundTripper.funcHandler.tsRoundTripperParams.maxRetries; i++ {
// set service url of target service of request only when
// trying to get new service url from cache/executor.
if retryCounter == 0 {
// get function service url from cache or executor
serviceUrl, serviceUrlFromCache, err = roundTripper.funcHandler.getServiceEntry()
roundTripper.serviceUrl, roundTripper.urlFromCache, err = roundTripper.funcHandler.getServiceEntry()
if err != nil {
// We might want a specific error code or header for fission failures as opposed to
// user function bugs.
@@ -228,21 +213,16 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
// service url maybe nil if router cannot find one in cache,
// so here we retry to get service url again
if serviceUrl == nil {
if roundTripper.serviceUrl == nil {
time.Sleep(executingTimeout)
executingTimeout = executingTimeout * time.Duration(roundTripper.funcHandler.tsRoundTripperParams.timeoutExponent)
continue
}
// tapService before invoking roundTrip for the serviceUrl
if serviceUrlFromCache {
go roundTripper.funcHandler.tapService(serviceUrl)
}
// modify the request to reflect the service url
// this service url may have come from the cache lookup or from executor response
req.URL.Scheme = serviceUrl.Scheme
req.URL.Host = serviceUrl.Host
req.URL.Scheme = roundTripper.serviceUrl.Scheme
req.URL.Host = roundTripper.serviceUrl.Host
// To keep the function run container simple, it
// doesn't do any routing. In the future if we have
@@ -254,7 +234,7 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
// Overwrite request host with internal host,
// or request will be blocked in some situations
// (e.g. istio-proxy)
req.Host = serviceUrl.Host
req.Host = roundTripper.serviceUrl.Host
}
// over-riding default settings.
@@ -263,37 +243,21 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
KeepAlive: roundTripper.funcHandler.tsRoundTripperParams.keepAliveTime,
}).DialContext
overhead := time.Since(startTime)
roundTripper.logger.Debug("request headers", zap.Any("headers", req.Header))
// Creating context for client
if roundTripper.timeout <= 0 {
roundTripper.timeout = fv1.DEFAULT_FUNCTION_TIMEOUT
}
roundTripper.logger.Debug("Creating context for request for ", zap.Any("time", roundTripper.timeout))
// pass request context as parent context for the case
// that user aborts connection before timeout. Otherwise,
// the request won't be canceled until the deadline exceeded
// which may be a potential security issue.
ctx, closeCtx := context.WithTimeout(req.Context(), time.Duration(roundTripper.timeout)*time.Second)
// Do NOT assign returned request to "req"
// because the request used in the last round
// will be canceled when calling setContext.
newReq := roundTripper.setContext(req)
// forward the request to the function service
resp, err = ocRoundTripper.RoundTrip(req.WithContext(ctx))
closeCtx()
resp, err := ocRoundTripper.RoundTrip(newReq)
if err == nil {
// Track metrics
httpMetricLabels.code = resp.StatusCode
funcMetricLabels.cached = serviceUrlFromCache
functionCallCompleted(funcMetricLabels, httpMetricLabels,
overhead, time.Since(startTime), resp.ContentLength)
// return response back to user
return resp, nil
} else if i >= roundTripper.funcHandler.tsRoundTripperParams.maxRetries-1 {
}
roundTripper.totalRetry += 1
if i >= roundTripper.funcHandler.tsRoundTripperParams.maxRetries-1 {
// return here if we are in the last round
roundTripper.logger.Error("error getting response from function",
zap.String("function_name", fnMeta.Name),
@@ -316,11 +280,16 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
return resp, err
}
// close response body before entering next loop
if resp != nil {
resp.Body.Close()
}
// Check whether an error is an timeout error ("dial tcp i/o timeout").
// If it's not a timeout error or retryCounter exceeded pre-defined threshold,
// we assume the entry in router cache is stale, invalidate it.
if !isNetTimeoutErr || retryCounter >= roundTripper.funcHandler.tsRoundTripperParams.svcAddrRetryCount {
if serviceUrlFromCache {
if roundTripper.urlFromCache {
// if transport.RoundTrip returns a network dial error and serviceUrl was from cache,
// it means, the entry in router cache is stale, so invalidate it.
roundTripper.logger.Debug("request errored out - removing function from router's cache and requesting a new service for function",
@@ -342,11 +311,6 @@ func (roundTripper RetryingRoundTripper) RoundTrip(req *http.Request) (*http.Res
roundTripper.logger.Debug("Backing off before retrying", zap.Any("backoff_time", executingTimeout), zap.Error(err))
time.Sleep(executingTimeout)
executingTimeout = executingTimeout * time.Duration(roundTripper.funcHandler.tsRoundTripperParams.timeoutExponent)
// close response body before entering next loop
if resp != nil {
resp.Body.Close()
}
}
e := errors.New("Unable to get service url for connection")
@@ -373,25 +337,47 @@ func (roundTripper RetryingRoundTripper) getDefaultTransport() *http.Transport {
}
}
func (fh *functionHandler) tapService(serviceUrl *url.URL) {
// setContext returns a shallow copy of request with a new timeout context.
func (roundTripper *RetryingRoundTripper) setContext(req *http.Request) *http.Request {
if roundTripper.closeContextFunc != nil {
(*roundTripper.closeContextFunc)()
}
// pass request context as parent context for the case
// that user aborts connection before timeout. Otherwise,
// the request won't be canceled until the deadline exceeded
// which may be a potential security issue.
ctx, closeCtx := context.WithTimeout(req.Context(), roundTripper.funcTimeout)
roundTripper.closeContextFunc = &closeCtx
return req.WithContext(ctx)
}
// closeContext closes the context to release resources.
func (roundTripper *RetryingRoundTripper) closeContext() {
if roundTripper.closeContextFunc != nil {
(*roundTripper.closeContextFunc)()
}
}
func (fh *functionHandler) tapService(fn *fv1.Function, serviceUrl *url.URL) {
if fh.executor == nil {
return
}
fh.executor.TapService(serviceUrl)
fh.executor.TapService(fn.Metadata, fn.Spec.InvokeStrategy.ExecutionStrategy.ExecutorType, serviceUrl)
}
func (fh functionHandler) handler(responseWriter http.ResponseWriter, request *http.Request) {
if fh.httpTrigger != nil && fh.httpTrigger.Spec.FunctionReference.Type == types.FunctionReferenceTypeFunctionWeights {
// canary deployment. need to determine the function to send request to now
fnMetadata := getCanaryBackend(fh.functionMetadataMap, fh.fnWeightDistributionList)
if fnMetadata == nil {
fn := getCanaryBackend(fh.functionMap, fh.fnWeightDistributionList)
if fn == nil {
fh.logger.Error("could not get canary backend",
zap.Any("metadataMap", fh.functionMetadataMap),
zap.Any("fnMap", fh.functionMap),
zap.Any("distributionList", fh.fnWeightDistributionList))
// TODO : write error to responseWrite and return response
return
}
fh.function = fnMetadata
fh.function = fn
fh.logger.Debug("chosen function backend's metadata", zap.Any("metadata", fh.function))
}
@@ -399,7 +385,7 @@ func (fh functionHandler) handler(responseWriter http.ResponseWriter, request *h
setPathInfoToHeader(request)
// system params
setFunctionMetadataToHeader(fh.function, request)
setFunctionMetadataToHeader(&fh.function.Metadata, request)
director := func(req *http.Request) {
if _, ok := req.Header["User-Agent"]; !ok {
@@ -408,21 +394,42 @@ func (fh functionHandler) handler(responseWriter http.ResponseWriter, request *h
}
}
var timeout int = fv1.DEFAULT_FUNCTION_TIMEOUT
if fh.functionTimeoutMap != nil {
timeout = fh.functionTimeoutMap[fh.function.GetUID()]
fnTimeout := fh.functionTimeoutMap[fh.function.Metadata.GetUID()]
if fnTimeout == 0 {
fnTimeout = fv1.DEFAULT_FUNCTION_TIMEOUT
}
proxy := &httputil.ReverseProxy{
Director: director,
Transport: &RetryingRoundTripper{
logger: fh.logger.Named("roundtripper"),
funcHandler: &fh,
timeout: timeout,
},
ErrorHandler: getProxyErrorHandler(fh.logger, fh.function),
rrt := &RetryingRoundTripper{
logger: fh.logger.Named("roundtripper"),
funcHandler: &fh,
funcTimeout: time.Duration(fnTimeout) * time.Second,
}
start := time.Now()
proxy := &httputil.ReverseProxy{
Director: director,
Transport: rrt,
ErrorHandler: fh.getProxyErrorHandler(start, rrt),
ModifyResponse: func(resp *http.Response) error {
go fh.collectFunctionMetric(start, rrt, request, resp)
return nil
},
}
defer func() {
// If the context is closed when RoundTrip returns, client may receive
// truncated response body due to "context canceled" error. To avoid
// this, we need to close request context after proxy.ServeHTTP finished.
//
// NOTE: rrt.closeContext() must be put in the defer function; otherwise,
// reverseProxy may panic when failed to write response and the context
// will not be closed.
//
// ref: https://github.com/golang/go/issues/28239
rrt.closeContext()
}()
proxy.ServeHTTP(responseWriter, request)
}
@@ -453,38 +460,10 @@ func findCeil(randomNumber int, wtDistrList []FunctionWeightDistribution) string
}
// picks a function to route to based on a random number generated
func getCanaryBackend(fnMetadatamap map[string]*metav1.ObjectMeta, fnWtDistributionList []FunctionWeightDistribution) *metav1.ObjectMeta {
func getCanaryBackend(fnMap map[string]*fv1.Function, fnWtDistributionList []FunctionWeightDistribution) *fv1.Function {
randomNumber := rand.Intn(fnWtDistributionList[len(fnWtDistributionList)-1].sumPrefix + 1)
fnName := findCeil(randomNumber, fnWtDistributionList)
return fnMetadatamap[fnName]
}
// getProxyErrorHandler returns a reverse proxy error handler
func getProxyErrorHandler(logger *zap.Logger, fnMeta *metav1.ObjectMeta) func(rw http.ResponseWriter, req *http.Request, err error) {
return func(rw http.ResponseWriter, req *http.Request, err error) {
status := http.StatusBadGateway
switch err {
case context.Canceled:
// 499 CLIENT CLOSED REQUEST
// A non-standard status code introduced by nginx for the case
// when a client closes the connection while nginx is processing the request.
// Reference: https://httpstatuses.com/499
status = 499
logger.Debug("client closes the connection",
zap.Any("function", fnMeta), zap.Any("request_header", req.Header))
case context.DeadlineExceeded:
status = http.StatusGatewayTimeout
logger.Error("function not responses before the timeout",
zap.Any("function", fnMeta), zap.Any("request_header", req.Header))
default:
logger.Error("error sending request to function",
zap.Error(err), zap.Any("function", fnMeta), zap.Any("request_header", req.Header))
}
// TODO: return error message that contains traceable UUID back to user. Issue #693
rw.WriteHeader(status)
}
return fnMap[fnName]
}
// addForwardedHostHeader add "forwarded host" to request header
@@ -544,28 +523,30 @@ func (fh *functionHandler) getServiceEntry() (serviceUrl *url.URL, serviceUrlFro
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
fnMeta := &fh.function.Metadata
// Use throttle to limit the total amount of requests sent
// to the executor to prevent it from overloaded.
recordObj, err := fh.svcAddrUpdateThrottler.RunOnce(
crd.CacheKey(fh.function),
crd.CacheKey(fnMeta),
func(firstToTheLock bool) (interface{}, error) {
var u *url.URL
// Get service entry from executor and update cache if its the first goroutine
if firstToTheLock { // first to the service url
fh.logger.Debug("calling getServiceForFunction",
zap.String("function_name", fh.function.Name))
zap.String("function_name", fnMeta.Name))
u, err = fh.getServiceEntryFromExecutor(ctx)
if err != nil {
fh.logger.Error("error getting service url from executor",
zap.Error(err),
zap.String("function_name", fh.function.Name))
zap.String("function_name", fnMeta.Name))
return nil, err
}
// add the address in router's cache
fh.logger.Info("assigning service url for function",
zap.String("url", u.String()),
zap.String("function_name", fh.function.Name))
fh.fmap.assign(fh.function, u)
zap.String("function_name", fnMeta.Name))
fh.fmap.assign(fnMeta, u)
} else {
u, err = fh.getServiceEntryFromCache()
if err != nil {
@@ -583,9 +564,9 @@ func (fh *functionHandler) getServiceEntry() (serviceUrl *url.URL, serviceUrlFro
e := "error updating service address entry for function"
fh.logger.Error(e,
zap.Error(err),
zap.String("function_name", fh.function.Name),
zap.String("function_namespace", fh.function.Namespace))
return nil, false, errors.Wrapf(err, "%s %s_%s", e, fh.function.Name, fh.function.Namespace)
zap.String("function_name", fnMeta.Name),
zap.String("function_namespace", fnMeta.Namespace))
return nil, false, errors.Wrapf(err, "%s %s_%s", e, fnMeta.Name, fnMeta.Namespace)
}
record, ok := recordObj.(svcEntryRecord)
@@ -597,9 +578,9 @@ func (fh *functionHandler) getServiceEntry() (serviceUrl *url.URL, serviceUrlFro
}
// getServiceEntryFromCache returns service url entry returns from cache
func (fh *functionHandler) getServiceEntryFromCache() (serviceUrl *url.URL, err error) {
func (fh functionHandler) getServiceEntryFromCache() (serviceUrl *url.URL, err error) {
// cache lookup to get serviceUrl
serviceUrl, err = fh.fmap.lookup(fh.function)
serviceUrl, err = fh.fmap.lookup(&fh.function.Metadata)
if err != nil {
var errMsg string
@@ -612,7 +593,7 @@ func (fh *functionHandler) getServiceEntryFromCache() (serviceUrl *url.URL, err
if e.Code == ferror.ErrorNotFound {
return nil, nil
}
errMsg = fmt.Sprintf("Error getting function %v;s service entry from cache: %v", fh.function.Name, err)
errMsg = fmt.Sprintf("Error getting function %v;s service entry from cache: %v", fh.function.Metadata.Name, err)
}
return nil, ferror.MakeError(http.StatusInternalServerError, errMsg)
}
@@ -620,9 +601,9 @@ func (fh *functionHandler) getServiceEntryFromCache() (serviceUrl *url.URL, err
}
// getServiceEntryFromExecutor returns service url entry returns from executor
func (fh *functionHandler) getServiceEntryFromExecutor(ctx context.Context) (*url.URL, error) {
func (fh functionHandler) getServiceEntryFromExecutor(ctx context.Context) (*url.URL, error) {
// send a request to executor to specialize a new pod
service, err := fh.executor.GetServiceForFunction(ctx, fh.function)
service, err := fh.executor.GetServiceForFunction(ctx, &fh.function.Metadata)
if err != nil {
statusCode, errMsg := ferror.GetHTTPError(err)
fh.logger.Error("error from GetServiceForFunction",
@@ -644,3 +625,72 @@ func (fh *functionHandler) getServiceEntryFromExecutor(ctx context.Context) (*ur
return serviceUrl, nil
}
// getProxyErrorHandler returns a reverse proxy error handler
func (fh functionHandler) getProxyErrorHandler(start time.Time, rrt *RetryingRoundTripper) func(rw http.ResponseWriter, req *http.Request, err error) {
return func(rw http.ResponseWriter, req *http.Request, err error) {
var status int
var msg string
switch err {
case context.Canceled:
// 499 CLIENT CLOSED REQUEST
// A non-standard status code introduced by nginx for the case
// when a client closes the connection while nginx is processing the request.
// Reference: https://httpstatuses.com/499
status = 499
msg = "client closes the connection"
fh.logger.Debug(msg, zap.Any("function", fh.function), zap.Any("request_header", req.Header))
case context.DeadlineExceeded:
status = http.StatusGatewayTimeout
msg = "function not responses before the timeout"
fh.logger.Error(msg, zap.Any("function", fh.function), zap.Any("request_header", req.Header))
default:
status = http.StatusBadGateway
msg = "error sending request to function"
fh.logger.Error(msg, zap.Error(err), zap.Any("function", fh.function), zap.Any("request_header", req.Header))
}
go fh.collectFunctionMetric(start, rrt, req, &http.Response{
StatusCode: status,
ContentLength: 0,
})
// TODO: return error message that contains traceable UUID back to user. Issue #693
rw.WriteHeader(status)
rw.Write([]byte(msg))
}
}
func (fh functionHandler) collectFunctionMetric(start time.Time, rrt *RetryingRoundTripper, req *http.Request, resp *http.Response) {
duration := time.Since(start)
// Metrics stuff
funcMetricLabels := &functionLabels{
namespace: fh.function.Metadata.Namespace,
name: fh.function.Metadata.Name,
}
httpMetricLabels := &httpLabels{
method: req.Method,
}
if fh.httpTrigger != nil {
httpMetricLabels.host = fh.httpTrigger.Spec.Host
httpMetricLabels.path = fh.httpTrigger.Spec.RelativeURL
}
// Track metrics
httpMetricLabels.code = resp.StatusCode
funcMetricLabels.cached = rrt.urlFromCache
functionCallCompleted(funcMetricLabels, httpMetricLabels,
duration, duration, resp.ContentLength)
// tapService before invoking roundTrip for the serviceUrl
if rrt.urlFromCache {
fh.tapService(fh.function, rrt.serviceUrl)
}
fh.logger.Debug("Request complete", zap.String("function", fh.function.Metadata.Name),
zap.Int("retry", rrt.totalRetry), zap.Duration("total-time", duration),
zap.Int64("content-length", resp.ContentLength))
}
+19 -6
View File
@@ -57,12 +57,12 @@ func TestFunctionProxying(t *testing.T) {
backendURL := createBackendService(testResponseString)
log.Printf("Created backend svc at %v", backendURL)
fn := &metav1.ObjectMeta{Name: "foo", Namespace: metav1.NamespaceDefault}
fnMeta := metav1.ObjectMeta{Name: "foo", Namespace: metav1.NamespaceDefault}
logger, err := zap.NewDevelopment()
panicIf(err)
fmap := makeFunctionServiceMap(logger, 0)
fmap.assign(fn, backendURL)
fmap.assign(&fnMeta, backendURL)
httpTrigger := &fv1.HTTPTrigger{
Metadata: metav1.ObjectMeta{
@@ -78,9 +78,11 @@ func TestFunctionProxying(t *testing.T) {
}
fh := &functionHandler{
logger: logger,
fmap: fmap,
function: fn,
logger: logger,
fmap: fmap,
function: &fv1.Function{
Metadata: metav1.ObjectMeta{Name: "foo", Namespace: metav1.NamespaceDefault},
},
tsRoundTripperParams: &tsRoundTripperParams{
timeout: 50 * time.Millisecond,
timeoutExponent: 2,
@@ -97,7 +99,18 @@ func TestFunctionProxying(t *testing.T) {
func TestProxyErrorHandler(t *testing.T) {
logger, err := zap.NewDevelopment()
assert.Nil(t, err)
errHandler := getProxyErrorHandler(logger, nil)
fh := &functionHandler{
logger: logger,
function: &fv1.Function{
Metadata: metav1.ObjectMeta{
Name: "dummy",
Namespace: "dummy-bar",
},
},
}
errHandler := fh.getProxyErrorHandler(time.Now(), &RetryingRoundTripper{})
req, err := http.NewRequest("GET", "http://foobar.com", nil)
assert.Nil(t, err)
+9 -9
View File
@@ -50,7 +50,7 @@ type (
// a distribution of requests across two functions.
resolveResult struct {
resolveResultType
functionMetadataMap map[string]*metav1.ObjectMeta
functionMap map[string]*fv1.Function
functionWtDistributionList []FunctionWeightDistribution
}
@@ -134,12 +134,13 @@ func (frr *functionReferenceResolver) resolveByName(namespace, name string) (*re
}
f := obj.(*fv1.Function)
functionMetadataMap := make(map[string]*metav1.ObjectMeta, 1)
functionMetadataMap[f.Metadata.Name] = &f.Metadata
functionMap := map[string]*fv1.Function{
f.Metadata.Name: f,
}
rr := resolveResult{
resolveResultType: resolveResultSingleFunction,
functionMetadataMap: functionMetadataMap,
resolveResultType: resolveResultSingleFunction,
functionMap: functionMap,
}
return &rr, nil
@@ -147,7 +148,7 @@ func (frr *functionReferenceResolver) resolveByName(namespace, name string) (*re
func (frr *functionReferenceResolver) resolveByFunctionWeights(namespace string, fr *fv1.FunctionReference) (*resolveResult, error) {
functionMetadataMap := make(map[string]*metav1.ObjectMeta)
functionMap := make(map[string]*fv1.Function)
fnWtDistrList := make([]FunctionWeightDistribution, 0)
sumPrefix := 0
@@ -167,19 +168,18 @@ func (frr *functionReferenceResolver) resolveByFunctionWeights(namespace string,
}
f := obj.(*fv1.Function)
functionMetadataMap[f.Metadata.Name] = &f.Metadata
functionMap[f.Metadata.Name] = f
sumPrefix = sumPrefix + functionWeight
fnWtDistrList = append(fnWtDistrList, FunctionWeightDistribution{
name: functionName,
weight: functionWeight,
sumPrefix: sumPrefix,
})
}
rr := resolveResult{
resolveResultType: resolveResultMultipleFunctions,
functionMetadataMap: functionMetadataMap,
functionMap: functionMap,
functionWtDistributionList: fnWtDistrList,
}
+10 -11
View File
@@ -142,7 +142,7 @@ func (ts *HTTPTriggerSet) getRouter(fnTimeoutMap map[types.UID]int) *mux.Router
fmap: ts.functionServiceMap,
executor: ts.executor,
httpTrigger: &trigger,
functionMetadataMap: rr.functionMetadataMap,
functionMap: rr.functionMap,
fnWeightDistributionList: rr.functionWtDistributionList,
tsRoundTripperParams: ts.tsRoundTripperParams,
isDebugEnv: ts.isDebugEnv,
@@ -158,8 +158,8 @@ func (ts *HTTPTriggerSet) getRouter(fnTimeoutMap map[types.UID]int) *mux.Router
// deployment. For more details, please check "handler" function of functionHandler.
if rr.resolveResultType == resolveResultSingleFunction {
for _, metadata := range fh.functionMetadataMap {
fh.function = metadata
for _, fn := range fh.functionMap {
fh.function = fn
}
}
@@ -185,20 +185,19 @@ func (ts *HTTPTriggerSet) getRouter(fnTimeoutMap map[types.UID]int) *mux.Router
// Internal triggers for each function by name. Non-http
// triggers route into these.
for _, function := range ts.functions {
m := function.Metadata
for i := range ts.functions {
fn := ts.functions[i]
fh := &functionHandler{
logger: ts.logger.Named(m.Name),
logger: ts.logger.Named(fn.Metadata.Name),
fmap: ts.functionServiceMap,
function: &m,
function: &fn,
executor: ts.executor,
tsRoundTripperParams: ts.tsRoundTripperParams,
isDebugEnv: ts.isDebugEnv,
svcAddrUpdateThrottler: ts.svcAddrUpdateThrottler,
functionTimeoutMap: fnTimeoutMap,
}
muxRouter.HandleFunc(utils.UrlForFunction(function.Metadata.Name, function.Metadata.Namespace), fh.handler)
muxRouter.HandleFunc(utils.UrlForFunction(fn.Metadata.Name, fn.Metadata.Namespace), fh.handler)
}
// Healthz endpoint for the router.
@@ -263,8 +262,8 @@ func (ts *HTTPTriggerSet) initFunctionController() (k8sCache.Store, k8sCache.Con
// update resolver function reference cache
for key, rr := range ts.resolver.copy() {
if key.namespace == fn.Metadata.Namespace &&
rr.functionMetadataMap[fn.Metadata.Name] != nil &&
rr.functionMetadataMap[fn.Metadata.Name].ResourceVersion != fn.Metadata.ResourceVersion {
rr.functionMap[fn.Metadata.Name] != nil &&
rr.functionMap[fn.Metadata.Name].Metadata.ResourceVersion != fn.Metadata.ResourceVersion {
// invalidate resolver cache
ts.logger.Debug("invalidating resolver cache")
err := ts.resolver.delete(key.namespace, key.triggerName, key.triggerResourceVersion)
+3
View File
@@ -71,6 +71,9 @@ func deleteIngress(logger *zap.Logger, trigger *fv1.HTTPTrigger, kubeClient *kub
}
func updateIngress(logger *zap.Logger, oldT *fv1.HTTPTrigger, newT *fv1.HTTPTrigger, kubeClient *kubernetes.Clientset) {
if !oldT.Spec.CreateIngress && !newT.Spec.CreateIngress {
return
}
if !oldT.Spec.CreateIngress && newT.Spec.CreateIngress {
createIngress(logger, newT, kubeClient)
+9 -7
View File
@@ -32,12 +32,12 @@ import (
func TestRouter(t *testing.T) {
// metadata for a fake function
fn := &metav1.ObjectMeta{Name: "foo", Namespace: metav1.NamespaceDefault}
fnMeta := metav1.ObjectMeta{Name: "foo", Namespace: metav1.NamespaceDefault}
// and a reference to it
fr := fv1.FunctionReference{
Type: types.FunctionReferenceTypeFunctionName,
Name: fn.Name,
Name: fnMeta.Name,
}
// start a fake service
@@ -49,7 +49,7 @@ func TestRouter(t *testing.T) {
// set up the cache with this fake service
fmap := makeFunctionServiceMap(logger, 0)
fmap.assign(fn, testServiceUrl)
fmap.assign(&fnMeta, testServiceUrl)
// HTTP trigger set with a trigger for this function
triggers, _, _ := makeHTTPTriggerSet(logger, fmap, nil, nil, nil, nil,
@@ -81,12 +81,14 @@ func TestRouter(t *testing.T) {
triggerResourceVersion: "1234",
}
fnMetaMap := make(map[string]*metav1.ObjectMeta, 1)
fnMetaMap[fn.Name] = fn
fnMetaMap := make(map[string]*fv1.Function, 1)
fnMetaMap[fnMeta.Name] = &fv1.Function{
Metadata: fnMeta,
}
rr := resolveResult{
resolveResultType: resolveResultSingleFunction,
functionMetadataMap: fnMetaMap,
resolveResultType: resolveResultSingleFunction,
functionMap: fnMetaMap,
}
frr.refCache.Set(nfr, rr)
+9 -13
View File
@@ -121,22 +121,18 @@ const (
// executor kubernetes object label key
const (
ENVIRONMENT_NAMESPACE = "environmentNamespace"
ENVIRONMENT_NAME = "environmentName"
ENVIRONMENT_UID = "environmentUid"
FUNCTION_NAMESPACE = "functionNamespace"
FUNCTION_NAME = "functionName"
FUNCTION_UID = "functionUid"
EXECUTOR_TYPE = "executorType"
ENVIRONMENT_NAMESPACE = "environmentNamespace"
ENVIRONMENT_NAME = "environmentName"
ENVIRONMENT_UID = "environmentUid"
FUNCTION_NAMESPACE = "functionNamespace"
FUNCTION_NAME = "functionName"
FUNCTION_UID = "functionUid"
FUNCTION_RESOURCE_VERSION = "functionResourceVersion"
EXECUTOR_TYPE = "executorType"
)
const (
ExecutorTypePoolmgr = fv1.ExecutorTypePoolmgr
ExecutorTypeNewdeploy = fv1.ExecutorTypeNewdeploy
)
const (
StrategyTypeExecution = fv1.StrategyTypeExecution
ANNOTATION_SVC_HOST = "svcHost"
)
const (
+35
View File
@@ -17,12 +17,15 @@ limitations under the License.
package utils
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"golang.org/x/net/context/ctxhttp"
"io"
"io/ioutil"
"net"
"net/http"
"os"
"path/filepath"
"strings"
@@ -198,3 +201,35 @@ func GetChecksum(src io.Reader) (*fv1.Checksum, error) {
func IsURL(str string) bool {
return strings.HasPrefix(str, "http://") || strings.HasPrefix(str, "https://")
}
func DownloadUrl(ctx context.Context, httpClient *http.Client, url string, localPath string) error {
resp, err := ctxhttp.Get(ctx, httpClient, url)
if err != nil {
return err
}
defer resp.Body.Close()
w, err := os.Create(localPath)
if err != nil {
return err
}
defer w.Close()
_, err = io.Copy(w, resp.Body)
if err != nil {
return err
}
// flushing write buffer to file
err = w.Sync()
if err != nil {
return err
}
err = os.Chmod(localPath, 0600)
if err != nil {
return err
}
return nil
}
+5 -3
View File
@@ -512,6 +512,7 @@ run_all_tests() {
export JOBS=6
$ROOT/test/run_test.sh \
$ROOT/test/tests/test_canary.sh \
$ROOT/test/tests/test_fn_update/test_idle_objects_reaper.sh \
$ROOT/test/tests/mqtrigger/kafka/test_kafka.sh \
$ROOT/test/tests/test_annotations.sh \
$ROOT/test/tests/test_archive_pruner.sh \
@@ -519,7 +520,6 @@ run_all_tests() {
$ROOT/test/tests/test_buildermgr.sh \
$ROOT/test/tests/test_env_vars.sh \
$ROOT/test/tests/test_environments/test_python_env.sh \
$ROOT/test/tests/test_fn_update/test_idle_objects_reaper.sh \
$ROOT/test/tests/test_function_test/test_fn_test.sh \
$ROOT/test/tests/test_function_update.sh \
$ROOT/test/tests/test_ingress.sh \
@@ -527,6 +527,7 @@ run_all_tests() {
$ROOT/test/tests/test_logging/test_function_logs.sh \
$ROOT/test/tests/test_node_hello_http.sh \
$ROOT/test/tests/test_package_command.sh \
$ROOT/test/tests/test_package_checksum.sh \
$ROOT/test/tests/test_pass.sh \
$ROOT/test/tests/test_router_cache_invalidation.sh \
$ROOT/test/tests/test_specs/test_spec.sh \
@@ -535,11 +536,12 @@ run_all_tests() {
$ROOT/test/tests/test_environments/test_tensorflow_serving_env.sh \
$ROOT/test/tests/test_environments/test_go_env.sh \
$ROOT/test/tests/mqtrigger/nats/test_mqtrigger.sh \
$ROOT/test/tests/mqtrigger/nats/test_mqtrigger_error.sh
$ROOT/test/tests/mqtrigger/nats/test_mqtrigger_error.sh \
$ROOT/test/tests/test_huge_response/test_huge_response.sh
FAILURES=$?
# FIXME: run tests with newdeploy one by one.
export JOBS=2
export JOBS=3
$ROOT/test/run_test.sh \
$ROOT/test/tests/test_backend_newdeploy.sh \
$ROOT/test/tests/test_environments/test_java_builder.sh \
@@ -41,7 +41,7 @@ timeout 60 bash -c "test_fn ${fn}-gpm 'world'"
log "Waiting for idle pod reaper to recycle resources"
# the LIST_OLD function list fsvc older than 2 mins
# so in worst case, we need to wait for up to 4 mins + some buffer
sleep 260
sleep 300
# The replicas of function deployment should be 0 due to minScale = 0
ndDeployReplicas=$(kubectl -n $FUNCTION_NAMESPACE get deploy -l functionName=${fn}-nd -ojsonpath='{.items[0].spec.replicas}')
File diff suppressed because it is too large Load Diff
+16
View File
@@ -0,0 +1,16 @@
package main
import (
"io/ioutil"
"net/http"
)
// Handler is the entry point for this fission function
func Handler(w http.ResponseWriter, r *http.Request) {
bytes, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Write(bytes)
}
+73
View File
@@ -0,0 +1,73 @@
#!/bin/bash
set -euo pipefail
source $(dirname $0)/../../utils.sh
TEST_ID=$(generate_test_id)
echo "TEST_ID = $TEST_ID"
ROOT=$(dirname $0)/../../..
pushd $ROOT/test/tests/test_huge_response
cleanup() {
clean_resource_by_id $TEST_ID
rm -rf response.json
popd
}
retryPost() {
local fn=$1
log "Send huge JSON request body"
set +e
while true; do
curl -X POST http://$FISSION_ROUTER/$fn \
-H "Content-Type: application/json" \
--data-binary "@generated.json" > response.json
difftext=$(diff generated.json response.json)
if [ -z "$difftext" ]; then
break
else
echo "Receive truncated body"
sleep 1
fi
done
set -e
}
export -f retryPost
if [ -z "${TEST_NOCLEANUP:-}" ]; then
trap cleanup EXIT
else
log "TEST_NOCLEANUP is set; not cleaning up test artifacts afterwards."
fi
env=go-$TEST_ID
fn_poolmgr=hello-go-poolmgr-$TEST_ID
log "Creating environment for Golang"
fission env create --name $env --image $GO_RUNTIME_IMAGE --builder $GO_BUILDER_IMAGE --period 5
timeout 90 bash -c "wait_for_builder $env"
pkgName=$(generate_test_id)
fission package create --name $pkgName --src hello.go --env $env
# wait for build to finish at most 90s
timeout 90 bash -c "waitBuild $pkgName"
log "Creating function for Golang"
fission fn create --name $fn_poolmgr --env $env --pkg $pkgName --entrypoint Handler
log "Creating route for function"
fission route create --name $fn_poolmgr --function $fn_poolmgr --url /$fn_poolmgr --method POST
log "Waiting for router & pools to catch up"
sleep 5
log "Testing function"
timeout 20 bash -c "retryPost $fn_poolmgr"
log "Test PASSED"
+112
View File
@@ -0,0 +1,112 @@
#!/bin/bash
set -euo pipefail
source $(dirname $0)/../utils.sh
TEST_ID=$(generate_test_id)
echo "TEST_ID = $TEST_ID"
tmp_dir="/tmp/test-$TEST_ID"
mkdir -p $tmp_dir
ROOT=$(dirname $0)/../..
checkpkgsum() {
local pkg=$1
local sum=$2
pkgsum=$(kubectl -n default get packages ${pkg} -o jsonpath='{.spec.deployment.checksum.sum}')
if [ "${sum}" != "${pkgsum}" ]; then
log "have different sha256 checksum: ${sum} vs. ${pkgsum}"
kubectl -n default get packages ${pkg} -o yaml
exit 1
fi
}
cleanup() {
log "Cleaning up..."
clean_resource_by_id $TEST_ID
rm -rf $tmp_dir
}
if [ -z "${TEST_NOCLEANUP:-}" ]; then
trap cleanup EXIT
else
log "TEST_NOCLEANUP is set; not cleaning up test artifacts afterwards."
fi
log "Download test script"
url1="https://raw.githubusercontent.com/fission/fission/master/examples/nodejs/hello.js"
url2="https://raw.githubusercontent.com/fission/fission/master/examples/nodejs/hello-callback.js"
wget ${url1}
sum=$(shasum -a 256 hello.js|cut -d' ' -f 1)
wget ${url2}
sum2=$(shasum -a 256 hello-callback.js|cut -d' ' -f 1)
env="nodejs-${TEST_ID}"
log "Function with file URL"
fn1="fn1-$TEST_ID"
fission env create --name ${env} --image $NODE_RUNTIME_IMAGE --period 5
fission fn create --name ${fn1} --env ${env} --code ${url1}
pkgname=$(kubectl -n default get functions ${fn1} -o jsonpath="{.spec.package.packageref.name}")
checkpkgsum ${pkgname} ${sum}
log "Creating route"
fission route create --name ${fn1} --function ${fn1} --url /${fn1} --method GET
sleep 3
timeout 60 bash -c "test_fn ${fn1} 'hello, world'"
log "Update function with file URL"
fission fn update --name ${fn1} --env ${env} --code ${url2}
pkgname=$(kubectl -n default get functions ${fn1} -o jsonpath="{.spec.package.packageref.name}")
checkpkgsum ${pkgname} ${sum2}
sleep 3
timeout 60 bash -c "test_fn ${fn1} 'Hello, world callback!'"
log "Function with file URL & checksum"
fn2="fn2-$TEST_ID"
fission fn create --name ${fn2} --env ${env} --code ${url1} --deploychecksum ${sum}
pkgname=$(kubectl -n default get functions ${fn2} -o jsonpath="{.spec.package.packageref.name}")
checkpkgsum ${pkgname} ${sum}
log "Creating route"
fission route create --name ${fn2} --function ${fn2} --url /${fn2} --method GET
timeout 60 bash -c "test_fn ${fn2} 'hello, world'"
log "Function with file URL & insecure"
fn3="fn3-$TEST_ID"
fission fn create --name ${fn3} --env ${env} --code ${url1} --insecure
pkgname=$(kubectl -n default get functions ${fn3} -o jsonpath="{.spec.package.packageref.name}")
checkpkgsum ${pkgname} ""
log "Creating route"
fission route create --name ${fn3} --function ${fn3} --url /${fn3} --method GET
sleep 3
timeout 60 bash -c "test_fn ${fn3} 'hello, world'"
pkg1="pkg1-$TEST_ID"
pkg2="pkg2-$TEST_ID"
pkg3="pkg3-$TEST_ID"
fission pkg create --name ${pkg1} --env ${env} --code ${url1}
checkpkgsum ${pkg1} ${sum}
fission pkg update --name ${pkg1} --env ${env} --code ${url2}
checkpkgsum ${pkg1} ${sum2}
fission pkg create --name ${pkg2} --env ${env} --code ${url1} --deploychecksum ${sum}
checkpkgsum ${pkg2} ${sum}
fission pkg create --name ${pkg3} --env ${env} --code ${url1} --insecure
checkpkgsum ${pkg3} ""
log "Test PASSED"
exit 0
@@ -20,7 +20,6 @@ pushd $(dirname $0)
fission spec apply
fission fn test --name $fn_p
fission fn test --name $fn_nd
hnd=$(kubectl -n $FUNCTION_NAMESPACE get deployment -l=functionName=$fn_nd -ojsonpath='{.items[0].spec.template.spec.hostname}')
+6 -2
View File
@@ -44,6 +44,10 @@ clean_resource_by_id() {
}
test_fn() {
if [ -z $FISSION_ROUTER ]; then
log "Environment FISSION_ROUTER not set"
exit 1
fi
url="http://$FISSION_ROUTER/$1"
expect=$2
test_response $url $expect
@@ -162,8 +166,8 @@ export FISSION_NATS_STREAMING_URL="http://defaultFissionAuthToken@$(kubectl -n $
## To change the environment image setting for CI test, please refer run_all_tests() in test_utils.sh.
export PYTHON_RUNTIME_IMAGE=${PYTHON_RUNTIME_IMAGE:-fission/python-env}
export PYTHON_BUILDER_IMAGE=${PYTHON_BUILDER_IMAGE:-fission/python-builder}
export GO_RUNTIME_IMAGE=${GO_RUNTIME_IMAGE:-fission/go-env}
export GO_BUILDER_IMAGE=${GO_BUILDER_IMAGE:-fission/go-builder}
export GO_RUNTIME_IMAGE=${GO_RUNTIME_IMAGE:-fission/go-env-1.12}
export GO_BUILDER_IMAGE=${GO_BUILDER_IMAGE:-fission/go-builder-1.12}
export JVM_RUNTIME_IMAGE=${JVM_RUNTIME_IMAGE:-fission/jvm-env}
export JVM_BUILDER_IMAGE=${JVM_BUILDER_IMAGE:-fission/jvm-builder}
export NODE_RUNTIME_IMAGE=${NODE_RUNTIME_IMAGE:-fission/node-env}