# Handoff: Fission poolmgr reuse still broken after partial fix

## Goal

Нужно разобраться, почему в live Fission при последовательных invoke одной и той же function с executor `poolmgr` executor каждый раз снова делает `choosePod + specializePod`, вместо reuse уже существующего specialized pod/function service.

Ключевой симптом:

- после завершения invoke resources не reuse-ятся для следующего вызова;
- executor продолжает создавать новый specialized pod почти на каждый invoke;
- для тяжелых функций это приводит к росту pod count и затем к timeout/504.

## Environment

- Local workspace: `/home/naeel/remote_dev/fission`
- Remote console repo: `/home/naeel/terra/fission`
- Remote Fission runtime repo: `/home/naeel/terra/fission-src`
- Live cluster access only via SSH:

```bash
ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 '...'
```

- Fission namespace: `fission`
- Live user namespace under test: `fission-ffd1f598c169b0ae`

## Functions under test

Namespace: `fission-ffd1f598c169b0ae`

- `phppp`
  - env: `console-php-env`
  - executor: `poolmgr`
  - route: `/f598c169b0ae/phppp`
- `88`
  - env: `console-python-env`
  - executor: `poolmgr`
  - route: `/f598c169b0ae/88`

HTTP invoke path used for live validation:

```bash
https://fission.kube5s.ru/fn/f598c169b0ae/phppp
https://fission.kube5s.ru/fn/f598c169b0ae/88
```

Valid user token used for direct invoke is in:

- `/home/naeel/remote_dev/terraform/TEST_STAND/POSTGRES/terraform.tfvars`
- remote mirror: `/home/naeel/terra/terraform/TEST_STAND/POSTGRES/terraform.tfvars`

## What was already found earlier

Изначально казалось, что проблема только в namespace quota saturation, потому что при многократных invoke возникали `504`, а quota по CPU/memory/pods реально упирались в потолок.

Но затем был найден реальный runtime bug:

- reuse key в executor uses `UID + ResourceVersion + Generation`
- `CacheKeyURGFromMeta(...)` in `pkg/crd/key.go`
- router/executor client path previously sent `FnMetadata` in `TapService`/`UnTapService` without `Generation`
- из-за этого `UnTapService -> MarkAvailable` не попадал в тот же cache key
- `activeRequests` не уменьшался у правильного cached function service
- следующий invoke видел cache miss и создавал новый specialized pod

### Previous fix already made

File changed in runtime repo:

- `pkg/executor/client/client.go`

Change:

- added `Generation: fnMeta.Generation` in `TapService(...)`
- added `Generation: fnMeta.Generation` in `UnTapService(...)`

Remote commit already made:

- `775845b executor: preserve generation in untap metadata`

## Important: this fix is already rolled out to live

New bundle image built and deployed:

- image: `naeel/fission-bundle:v1.22.0-multi-ns-9`
- digest: `sha256:5348bca2eacb78a030e639d7f4f7ac392369b6f702677ac47a69ea9f61f50a12`

Live deployment state after rollout:

- `router` image = `naeel/fission-bundle:v1.22.0-multi-ns-9`
- `executor` image = `naeel/fission-bundle:v1.22.0-multi-ns-9`

So: the `Generation` fix is **already in live**, but reuse is **still broken**.

## Current live reproduction

### `phppp`

Command pattern:

```bash
USER_TOKEN=$(grep -m1 '^api_token' /home/naeel/terra/terraform/TEST_STAND/POSTGRES/terraform.tfvars | sed -E 's/^api_token = "(.*)"$/\1/')
curl -sk -o /tmp/phppp.out -w '%{http_code}' --max-time 40 \
  "https://fission.kube5s.ru/fn/f598c169b0ae/phppp" \
  -H "Authorization: Bearer $USER_TOKEN"
```

Observed result:

- 10 of 10 invokes returned `200`
- BUT php specialized pod count grew from `1` to `11`

Exact observed summary:

- `PHP_PODS_BEFORE=1`
- invokes `1..10`: all `HTTP=200`
- `PHP_PODS_AFTER=11`

This proves:

- invoke itself works
- auth token is valid
- runtime is still specializing a new pod on each invoke instead of reusing

### `88`

Command pattern:

```bash
USER_TOKEN=$(grep -m1 '^api_token' /home/naeel/terra/terraform/TEST_STAND/POSTGRES/terraform.tfvars | sed -E 's/^api_token = "(.*)"$/\1/')
curl -sk -o /tmp/88.out -w '%{http_code}' --max-time 40 \
  "https://fission.kube5s.ru/fn/f598c169b0ae/88" \
  -H "Authorization: Bearer $USER_TOKEN"
```

Observed result:

- first 3 invokes returned `200`
- then invokes 4..10 returned `000` / timeout-like failure from curl
- python pod count grew from `1` to `4`

Exact observed summary:

- `PY_PODS_BEFORE=1`
- tries 1..3: `HTTP=200`
- tries 4..10: `HTTP=000`
- `PY_PODS_AFTER=4`

This suggests:

- same reuse bug path still exists for python too
- when function is heavier, pod growth quickly returns the original operational problem

## Most important executor logs

After rollout, executor logs still show for `phppp` on nearly every invoke:

- `choosing pod from pool`
- `relabel pod`
- `specializing pod`
- `added function service`

Example pattern repeated for each invoke:

```text
choosing pod from pool
relabel pod
chose pod ... poolmgr-console-php-env-...-5dcc862z4hm
calling fetcher to copy function
specializing pod
specialized pod ... podIP=172.16.1.80
added function service ... serviceHost=172.16.1.80:8888

choosing pod from pool
relabel pod
chose pod ... poolmgr-console-php-env-...-5dcc86k2zlk
calling fetcher to copy function
specializing pod
specialized pod ... podIP=172.16.3.209
added function service ... serviceHost=172.16.3.209:8888

choosing pod from pool
relabel pod
chose pod ... poolmgr-console-php-env-...-5dcc86ftzfp
calling fetcher to copy function
specializing pod
specialized pod ... podIP=172.16.3.52
added function service ... serviceHost=172.16.3.52:8888
```

That is, executor behavior in live clearly remains:

- no effective reuse of existing function service
- no steady-state one-pod serving repeated invokes

Also later there is only idle cleanup like:

```text
release idle function resources
function=phppp
address=172.16.2.155:8888
```

That confirms pods are being reaped later, but not reused immediately for next request.

## What is already ruled out

### Not an auth issue

- token from `terraform.tfvars` is valid
- direct `/fn/...` invoke works with bearer token

### Not just quota

- quota pressure was real earlier
- but now we have a cleaner reproduction where repeated invoke still creates new specialized pod even before total failure
- for `phppp`, 10 requests succeed but pod count still grows `1 -> 11`

### Not only missing `Generation` anymore

- `Generation` was indeed missing before
- that fix is already committed, built, deployed
- live logs prove reuse is still broken after that fix

## Highest-value question now

Where else in the path does executor fail to map the completion of request back to the existing cached function service?

In other words: why does `UnTapService -> MarkAvailable` still not make the current function service reusable for the next invoke?

## Narrow code path to inspect

Please focus on the smallest path controlling release/reuse:

- `pkg/router/functionHandler.go`
  - request lifecycle
  - `getServiceEntry`
  - `unTapService`
  - what metadata and service URL/address are sent back after request completes
- `pkg/executor/client/client.go`
  - `TapService(...)`
  - `UnTapService(...)`
- `pkg/executor/api.go`
  - `/v2/getServiceForFunction`
  - `/v2/unTapService`
  - `/v2/tapServices`
- `pkg/executor/fscache/functionServiceCache.go`
  - `GetFuncSvc`
  - `AddFunc`
  - `MarkAvailable`
  - `TouchByAddress`
- `pkg/executor/fscache/poolcache.go`
  - activeRequests accounting
  - `markAvailable`
- `pkg/executor/executortype/poolmgr/gp.go`
  - `getFuncSvc`
  - `choosePod`
  - `specializePod`
- `pkg/executor/executortype/poolmgr/gpm.go`
  - `TapService`
  - `UnTapService`
  - idle reaper logic
- `pkg/crd/key.go`
  - `CacheKeyURGFromMeta`

## Strong suspicion

There is still a mismatch in at least one of these dimensions:

1. metadata key used by cache lookup / mark available
2. service address format used for matching existing function service
3. path by which router reports completion back to executor
4. activeRequests decrement path in pool cache
5. tap/untap batching behavior versus per-request lifecycle

### Concrete symptoms that support this

- every next invoke still enters `choosePod`
- `added function service` repeats with new pod IP every time
- there is no visible log evidence that the just-used address became available in time for the next invoke

## Helpful concrete questions for investigation

1. Does router send the same logical function identity on `getServiceForFunction` and `unTapService` after the `Generation` fix, or is there still some field mismatch?
2. Is `serviceURL`/address normalized differently between tap and untap, so the address cannot match the cached function service?
3. Does `MarkAvailable` expect `serviceHost` without scheme while router sends full URL or vice versa?
4. Is `activeRequests` decremented on the same object/key that `GetFuncSvc` checks on the next invoke?
5. Could `TouchByAddress` / `MarkAvailable` be operating on a stale or different cache entry than `AddFunc` created?
6. Is the router definitely calling untap on successful invoke for these direct `/fn/...` requests?
7. Is there any race where the next invoke starts before previous untap is processed, making this appear as no reuse even for sequential calls with ~4-5s spacing?
8. For `phppp`, since calls are sequential and each lasts ~4s, why is a new pod still chosen 5 seconds later if previous invoke already completed?

## Useful live commands already used

Check live deployment images:

```bash
ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 \
  'kubectl describe deploy router -n fission | sed -n "/Containers:/,/Conditions:/p"'

ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 \
  'kubectl describe deploy executor -n fission | sed -n "/Containers:/,/Conditions:/p"'
```

Check function and trigger:

```bash
ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 \
  'kubectl get function phppp -n fission-ffd1f598c169b0ae -o yaml | sed -n "1,220p"'

ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 \
  'kubectl get httptrigger phppp-route -n fission-ffd1f598c169b0ae -o yaml | sed -n "1,220p"'
```

Live smoke used to prove bug remains:

```bash
ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 <<'SSH'
set -e
USER_TOKEN=$(grep -m1 '^api_token' /home/naeel/terra/terraform/TEST_STAND/POSTGRES/terraform.tfvars | sed -E 's/^api_token = "(.*)"$/\1/')
NS=fission-ffd1f598c169b0ae
BEFORE=$(kubectl get pods -n "$NS" --no-headers | grep 'poolmgr-console-php-env' | wc -l)
echo "PHP_PODS_BEFORE=$BEFORE"
for i in 1 2 3 4 5 6 7 8 9 10; do
  code=$(curl -sk -o /tmp/phppp.out -w '%{http_code}' --max-time 40 "https://fission.kube5s.ru/fn/f598c169b0ae/phppp" -H "Authorization: Bearer $USER_TOKEN" || true)
  body=$(tr '\n' ' ' </tmp/phppp.out 2>/dev/null | head -c 200 || true)
  echo "TRY=$i HTTP=$code BODY=$body"
done
AFTER=$(kubectl get pods -n "$NS" --no-headers | grep 'poolmgr-console-php-env' | wc -l)
echo "PHP_PODS_AFTER=$AFTER"
SSH
```

Executor logs:

```bash
ssh -i ~/.ssh/naeel_vm_id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 naeel@5.172.178.213 \
  'kubectl logs -n fission deploy/executor --since=15m | egrep "phppp| 88 |functionName=88|choosing pod from pool|specializing pod|added function service|marking function service|MarkAvailable|tapService|unTapService" | tail -n 240'
```

## What is wanted from Sonnet

Не просто общая гипотеза, а конкретно:

1. pinpoint the next exact root cause in code
2. explain why the already-deployed `Generation` fix was necessary but insufficient
3. propose the smallest correct patch
4. say exactly how to validate that patch in live

Best outcome:

- a minimal patch in one or a few files
- explanation of the failing cache/release path
- exact reason repeated invokes still go through `choosePod` after success of prior invoke