Nail/fission-console
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a2e1723e3301cbdeb926540b8afefbf1a8ae8a2d
fission-console/doc/SONNET_HANDOFF_2026-04-27_POOLMGR_REUSE.md
T

12 KiB
Raw Blame History

Handoff: Fission poolmgr reuse still broken after partial fix

Goal

Нужно разобраться, почему в live Fission при последовательных invoke одной и той же function с executor poolmgr executor каждый раз снова делает choosePod + specializePod, вместо reuse уже существующего specialized pod/function service.

Ключевой симптом:

  • после завершения invoke resources не reuse-ятся для следующего вызова;
  • executor продолжает создавать новый specialized pod почти на каждый invoke;
  • для тяжелых функций это приводит к росту pod count и затем к timeout/504.

Environment

  • Local workspace: /home/naeel/remote_dev/fission
  • Remote console repo: /home/naeel/terra/fission
  • Remote Fission runtime repo: /home/naeel/terra/fission-src
  • Live cluster access only via SSH:
ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 '...'
  • Fission namespace: fission
  • Live user namespace under test: fission-ffd1f598c169b0ae

Functions under test

Namespace: fission-ffd1f598c169b0ae

  • phppp
    • env: console-php-env
    • executor: poolmgr
    • route: /f598c169b0ae/phppp
  • 88
    • env: console-python-env
    • executor: poolmgr
    • route: /f598c169b0ae/88

HTTP invoke path used for live validation:

https://fission.kube5s.ru/fn/f598c169b0ae/phppp
https://fission.kube5s.ru/fn/f598c169b0ae/88

Valid user token used for direct invoke is in:

  • /home/naeel/remote_dev/terraform/TEST_STAND/POSTGRES/terraform.tfvars
  • remote mirror: /home/naeel/terra/terraform/TEST_STAND/POSTGRES/terraform.tfvars

What was already found earlier

Изначально казалось, что проблема только в namespace quota saturation, потому что при многократных invoke возникали 504, а quota по CPU/memory/pods реально упирались в потолок.

Но затем был найден реальный runtime bug:

  • reuse key в executor uses UID + ResourceVersion + Generation
  • CacheKeyURGFromMeta(...) in pkg/crd/key.go
  • router/executor client path previously sent FnMetadata in TapService/UnTapService without Generation
  • из-за этого UnTapService -> MarkAvailable не попадал в тот же cache key
  • activeRequests не уменьшался у правильного cached function service
  • следующий invoke видел cache miss и создавал новый specialized pod

Previous fix already made

File changed in runtime repo:

  • pkg/executor/client/client.go

Change:

  • added Generation: fnMeta.Generation in TapService(...)
  • added Generation: fnMeta.Generation in UnTapService(...)

Remote commit already made:

  • 775845b executor: preserve generation in untap metadata

Important: this fix is already rolled out to live

New bundle image built and deployed:

  • image: naeel/fission-bundle:v1.22.0-multi-ns-9
  • digest: sha256:5348bca2eacb78a030e639d7f4f7ac392369b6f702677ac47a69ea9f61f50a12

Live deployment state after rollout:

  • router image = naeel/fission-bundle:v1.22.0-multi-ns-9
  • executor image = naeel/fission-bundle:v1.22.0-multi-ns-9

So: the Generation fix is already in live, but reuse is still broken.

Current live reproduction

phppp

Command pattern:

USER_TOKEN=$(grep -m1 '^api_token' /home/naeel/terra/terraform/TEST_STAND/POSTGRES/terraform.tfvars | sed -E 's/^api_token = "(.*)"$/\1/')
curl -sk -o /tmp/phppp.out -w '%{http_code}' --max-time 40 \
  "https://fission.kube5s.ru/fn/f598c169b0ae/phppp" \
  -H "Authorization: Bearer $USER_TOKEN"

Observed result:

  • 10 of 10 invokes returned 200
  • BUT php specialized pod count grew from 1 to 11

Exact observed summary:

  • PHP_PODS_BEFORE=1
  • invokes 1..10: all HTTP=200
  • PHP_PODS_AFTER=11

This proves:

  • invoke itself works
  • auth token is valid
  • runtime is still specializing a new pod on each invoke instead of reusing

88

Command pattern:

USER_TOKEN=$(grep -m1 '^api_token' /home/naeel/terra/terraform/TEST_STAND/POSTGRES/terraform.tfvars | sed -E 's/^api_token = "(.*)"$/\1/')
curl -sk -o /tmp/88.out -w '%{http_code}' --max-time 40 \
  "https://fission.kube5s.ru/fn/f598c169b0ae/88" \
  -H "Authorization: Bearer $USER_TOKEN"

Observed result:

  • first 3 invokes returned 200
  • then invokes 4..10 returned 000 / timeout-like failure from curl
  • python pod count grew from 1 to 4

Exact observed summary:

  • PY_PODS_BEFORE=1
  • tries 1..3: HTTP=200
  • tries 4..10: HTTP=000
  • PY_PODS_AFTER=4

This suggests:

  • same reuse bug path still exists for python too
  • when function is heavier, pod growth quickly returns the original operational problem

Most important executor logs

After rollout, executor logs still show for phppp on nearly every invoke:

  • choosing pod from pool
  • relabel pod
  • specializing pod
  • added function service

Example pattern repeated for each invoke:

choosing pod from pool
relabel pod
chose pod ... poolmgr-console-php-env-...-5dcc862z4hm
calling fetcher to copy function
specializing pod
specialized pod ... podIP=172.16.1.80
added function service ... serviceHost=172.16.1.80:8888

choosing pod from pool
relabel pod
chose pod ... poolmgr-console-php-env-...-5dcc86k2zlk
calling fetcher to copy function
specializing pod
specialized pod ... podIP=172.16.3.209
added function service ... serviceHost=172.16.3.209:8888

choosing pod from pool
relabel pod
chose pod ... poolmgr-console-php-env-...-5dcc86ftzfp
calling fetcher to copy function
specializing pod
specialized pod ... podIP=172.16.3.52
added function service ... serviceHost=172.16.3.52:8888

That is, executor behavior in live clearly remains:

  • no effective reuse of existing function service
  • no steady-state one-pod serving repeated invokes

Also later there is only idle cleanup like:

release idle function resources
function=phppp
address=172.16.2.155:8888

That confirms pods are being reaped later, but not reused immediately for next request.

What is already ruled out

Not an auth issue

  • token from terraform.tfvars is valid
  • direct /fn/... invoke works with bearer token

Not just quota

  • quota pressure was real earlier
  • but now we have a cleaner reproduction where repeated invoke still creates new specialized pod even before total failure
  • for phppp, 10 requests succeed but pod count still grows 1 -> 11

Not only missing Generation anymore

  • Generation was indeed missing before
  • that fix is already committed, built, deployed
  • live logs prove reuse is still broken after that fix

Highest-value question now

Where else in the path does executor fail to map the completion of request back to the existing cached function service?

In other words: why does UnTapService -> MarkAvailable still not make the current function service reusable for the next invoke?

Narrow code path to inspect

Please focus on the smallest path controlling release/reuse:

  • pkg/router/functionHandler.go
    • request lifecycle
    • getServiceEntry
    • unTapService
    • what metadata and service URL/address are sent back after request completes
  • pkg/executor/client/client.go
    • TapService(...)
    • UnTapService(...)
  • pkg/executor/api.go
    • /v2/getServiceForFunction
    • /v2/unTapService
    • /v2/tapServices
  • pkg/executor/fscache/functionServiceCache.go
    • GetFuncSvc
    • AddFunc
    • MarkAvailable
    • TouchByAddress
  • pkg/executor/fscache/poolcache.go
    • activeRequests accounting
    • markAvailable
  • pkg/executor/executortype/poolmgr/gp.go
    • getFuncSvc
    • choosePod
    • specializePod
  • pkg/executor/executortype/poolmgr/gpm.go
    • TapService
    • UnTapService
    • idle reaper logic
  • pkg/crd/key.go
    • CacheKeyURGFromMeta

Strong suspicion

There is still a mismatch in at least one of these dimensions:

  1. metadata key used by cache lookup / mark available
  2. service address format used for matching existing function service
  3. path by which router reports completion back to executor
  4. activeRequests decrement path in pool cache
  5. tap/untap batching behavior versus per-request lifecycle

Concrete symptoms that support this

  • every next invoke still enters choosePod
  • added function service repeats with new pod IP every time
  • there is no visible log evidence that the just-used address became available in time for the next invoke

Helpful concrete questions for investigation

  1. Does router send the same logical function identity on getServiceForFunction and unTapService after the Generation fix, or is there still some field mismatch?
  2. Is serviceURL/address normalized differently between tap and untap, so the address cannot match the cached function service?
  3. Does MarkAvailable expect serviceHost without scheme while router sends full URL or vice versa?
  4. Is activeRequests decremented on the same object/key that GetFuncSvc checks on the next invoke?
  5. Could TouchByAddress / MarkAvailable be operating on a stale or different cache entry than AddFunc created?
  6. Is the router definitely calling untap on successful invoke for these direct /fn/... requests?
  7. Is there any race where the next invoke starts before previous untap is processed, making this appear as no reuse even for sequential calls with ~4-5s spacing?
  8. For phppp, since calls are sequential and each lasts ~4s, why is a new pod still chosen 5 seconds later if previous invoke already completed?

Useful live commands already used

Check live deployment images:

ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 \
  'kubectl describe deploy router -n fission | sed -n "/Containers:/,/Conditions:/p"'

ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 \
  'kubectl describe deploy executor -n fission | sed -n "/Containers:/,/Conditions:/p"'

Check function and trigger:

ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 \
  'kubectl get function phppp -n fission-ffd1f598c169b0ae -o yaml | sed -n "1,220p"'

ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 \
  'kubectl get httptrigger phppp-route -n fission-ffd1f598c169b0ae -o yaml | sed -n "1,220p"'

Live smoke used to prove bug remains:

ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 <<'SSH'
set -e
USER_TOKEN=$(grep -m1 '^api_token' /home/naeel/terra/terraform/TEST_STAND/POSTGRES/terraform.tfvars | sed -E 's/^api_token = "(.*)"$/\1/')
NS=fission-ffd1f598c169b0ae
BEFORE=$(kubectl get pods -n "$NS" --no-headers | grep 'poolmgr-console-php-env' | wc -l)
echo "PHP_PODS_BEFORE=$BEFORE"
for i in 1 2 3 4 5 6 7 8 9 10; do
  code=$(curl -sk -o /tmp/phppp.out -w '%{http_code}' --max-time 40 "https://fission.kube5s.ru/fn/f598c169b0ae/phppp" -H "Authorization: Bearer $USER_TOKEN" || true)
  body=$(tr '\n' ' ' </tmp/phppp.out 2>/dev/null | head -c 200 || true)
  echo "TRY=$i HTTP=$code BODY=$body"
done
AFTER=$(kubectl get pods -n "$NS" --no-headers | grep 'poolmgr-console-php-env' | wc -l)
echo "PHP_PODS_AFTER=$AFTER"
SSH

Executor logs:

ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 \
  'kubectl logs -n fission deploy/executor --since=15m | egrep "phppp| 88 |functionName=88|choosing pod from pool|specializing pod|added function service|marking function service|MarkAvailable|tapService|unTapService" | tail -n 240'

What is wanted from Sonnet

Не просто общая гипотеза, а конкретно:

  1. pinpoint the next exact root cause in code
  2. explain why the already-deployed Generation fix was necessary but insufficient
  3. propose the smallest correct patch
  4. say exactly how to validate that patch in live

Best outcome:

  • a minimal patch in one or a few files
  • explanation of the failing cache/release path
  • exact reason repeated invokes still go through choosePod after success of prior invoke
Reference in New Issue View Git Blame Copy Permalink