# Fission Multi-Tenant Porting Guide

> **Purpose:** This document exists so that any AI agent or engineer can fully
> understand what was changed to add multi-tenancy to this Fission fork, why
> each decision was made, and what needs to be ported when a new upstream
> Fission release arrives.
>
> Fork base: `github.com/fission/fission` tag `v1.22.0` (2025-12-16)
> Our branch: `feature/multitenant`

---

## 1. The Problem We Solved

In stock Fission v1.22.0 all resource namespaces must be listed in the
`FISSION_RESOURCE_NAMESPACES` environment variable **before** the process starts.
Adding a new tenant namespace requires:

1. Patching that env var on executor, router, buildermgr deployments
2. Triggering a rolling restart of all three components (~30 s downtime each)

At scale (hundreds of tenants created continuously) this causes a permanent
rolling-restart loop and cascading failures for existing users.

**Our solution:** hot namespace registration without pod restarts. Any platform
(console, operator, CI/CD) creates a Kubernetes Namespace with label
`fission.io/managed=true` — all three Fission components detect it within
milliseconds via k8s Watch and register it live.

---

## 2. Integration Contract (external platforms)

The entire contract between an external platform and Fission is a single label:

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-abc123
  labels:
    fission.io/managed: "true"
```

No other coupling to Fission internals is required.

To remove a tenant namespace: delete the namespace or remove the label.
Current removal strategy is `track-only` (the manager records the event but does
not actively deregister — the executor types stop receiving events for deleted
resources naturally). `dispatch-remove` strategy exists in the model but is not
wired by default (see §8).

---

## 2a. How the Console (`../fission`) Integrates

The Fission Console (`github.com/naeel/fission`, package `console`) creates tenant
namespaces via `SetupFissionNamespace()` in
`console/internal/fission/namespace.go`.

That function does three things:
1. Creates the Namespace with two labels:
   - `managed-by=fission-console` — console's own filter
   - **`fission.io/managed=true`** — this is the NSWatcher trigger
2. Creates `fission-fetcher` and `fission-builder` ServiceAccounts in the new NS
3. Creates RoleBindings for all Fission system SAs (`fission-executor`,
   `fission-router`, `fission-buildermgr`, etc.) using `cluster-admin` scoped to
   the namespace

**The coupling is exactly one label.** The console does not call any Fission
internal API to register the namespace — it just sets `fission.io/managed=true`
and the NSWatcher in executor/router/buildermgr picks it up automatically within
~50ms.

**Before `v1.22.0-mt1` (today's deploy):** the console set the label but the
executor was running the official `ghcr.io/fission/fission-bundle:v1.22.0` image
which has no NSWatcher — so the label was silently ignored. Tenant namespaces
still worked because the console also created the SA/RoleBindings manually (step 2
and 3 above), so pool pods could start. But executor/router/buildermgr were not
dynamically aware of new namespaces — they relied on whatever was in
`FISSION_RESOURCE_NAMESPACES` at startup.

**After `v1.22.0-mt1`:** executor/router/buildermgr detect the label
automatically. The SA creation in `EnsureNamespaceSA` (our code in
`pkg/utils/serviceaccount.go`) now runs from the executor side as well — but since
the console already created the SA, `EnsureNamespaceSA` is a no-op (idempotent).
No conflict, no double work.

---

## 3. Backward Compatibility

`FISSION_RESOURCE_NAMESPACES` continues to work exactly as before. Namespaces
listed there are bootstrapped at startup with source `env` and do not require the
label. The NSWatcher layer adds **on top** of the existing mechanism — nothing
was removed.

---

## 4. File Map

### New files (did not exist in v1.22.0)

| File | Purpose | What breaks if removed |
|------|---------|------------------------|
| `pkg/utils/namespace_manager_model.go` | All types: `NamespaceRecord`, `NamespacePhase`, `NamespaceSource`, `NamespaceEvent`, `NamespaceRemovalStrategy`, `ManagedNamespaceWatcherConfig` | Everything — all other files import these types |
| `pkg/utils/namespace_manager.go` | `NamespaceManager` interface + `inMemoryNamespaceManager` implementation. `RunManagedNamespaceWatcher()` — the single entry point used by all three components. `NewNamespaceWatcherEventHandlers()` — k8s informer callbacks. `EnsureNamespaceSA` helper call site. | All NSWatcher functionality |
| `pkg/utils/namespace_manager_test.go` | Unit + integration tests for NamespaceManager | Tests only |
| `pkg/utils/namespace_manager_model_test.go` | Tests for model helpers | Tests only |
| `pkg/utils/serviceaccount.go` (was modified, `EnsureNamespaceSA` added at bottom) | `EnsureNamespaceSA(ctx, client, logger, ns)` — creates fission-fetcher SA/Role/RoleBinding in a new namespace idempotently | Pool pods in new namespaces fail to start (no SA to run fetcher) |
| `pkg/executor/multitenant/ns_watcher.go` | `StartNSWatcher()` — executor entry point. `registerNamespace()` — calls `AddNamespace` on global resolver + `EnsureNamespaceSA` + all executor types. | Executor never learns about new namespaces |
| `pkg/executor/multitenant/namespace_subscriber.go` | `NewNamespaceSubscriber()` — adapter from `NamespaceSubscriber` interface to executor `registerNamespace()` | Same as above |
| `pkg/executor/multitenant/ns_watcher_test.go` | Tests | Tests only |
| `pkg/executor/multitenant/namespace_subscriber_test.go` | Tests | Tests only |
| `pkg/router/ns_watcher.go` | `StartNSWatcher()` — router entry point, 5 lines | Router never learns about new namespaces |
| `pkg/router/namespace_subscriber.go` | `NewNamespaceSubscriber()` — adapter calling `HTTPTriggerSet.AddNamespace` | Same as above |
| `pkg/router/namespace_subscriber_test.go` | Tests | Tests only |
| `pkg/buildermgr/ns_watcher.go` | `StartNSWatcher()` — buildermgr entry point, 5 lines | Buildermgr never learns about new namespaces |
| `pkg/buildermgr/namespace_subscriber.go` | `NewNamespaceSubscriber()` — adapter calling `envWatcher.AddNamespace` + `pkgWatcher.AddNamespace` | Same as above |
| `pkg/buildermgr/namespace_subscriber_test.go` | Tests | Tests only |
| `deploy/multitenant/rbac.yaml` | ClusterRoles + ClusterRoleBindings for all three components (see §6) | Components crash at startup or fail to watch namespaces |

### Modified files (existed in v1.22.0, we changed them)

| File | What we added | What breaks if reverted |
|------|--------------|-------------------------|
| `pkg/utils/namespace.go` | `ManagedNamespaceLabelKey/Value` constants, `ManagedNamespaceLabelSelector()`, `IsManagedNamespace()`, `AddNamespace()` (thread-safe dedup), `Snapshot()` (sorted slice copy under read-lock), `SnapshotWithOptions()` | All callers of `Snapshot()` break — there are many; label constants used by informer filter |
| `pkg/utils/namespace_test.go` | Tests for new methods | Tests only |
| `pkg/utils/informer.go` | `NewSharedInformerFactoryForNamespaces(namespaces []string)` — creates informer factory filtered to a dynamic list of namespaces | Executor types cannot create per-NS informers for new namespaces |
| `pkg/executor/executor.go` | `StartNSWatcher(...)` call added after executor types are initialized | NSWatcher never starts in executor |
| `pkg/executor/executortype/executortype.go` | `AddNamespace(ctx, ns, mgr)` added to the `ExecutorType` interface | All three executor types must implement this; compilation fails |
| `pkg/executor/executortype/poolmgr/gpm.go` | `AddNamespace()` implementation — creates per-NS informer factory, pod lister, event handlers | poolmgr never picks up functions in new namespaces |
| `pkg/executor/executortype/poolmgr/poolpodcontroller.go` | Uses `Snapshot()` in runtime loop instead of static namespace list | Pool pods not created in new namespaces |
| `pkg/executor/executortype/newdeploy/newdeploymgr.go` | `AddNamespace()` implementation | newdeploy never picks up functions in new namespaces |
| `pkg/executor/executortype/container/containermgr.go` | `AddNamespace()` implementation | container executor never picks up functions in new namespaces |
| `pkg/router/router.go` | `StartNSWatcher(...)` call added | NSWatcher never starts in router |
| `pkg/router/httpTriggers.go` | `AddNamespace(ns string)` on `HTTPTriggerSet` — starts per-NS informers for HTTPTriggers and Functions | Router ignores HTTPTriggers in new namespaces |
| `pkg/router/functionReferenceResolver.go` | Uses `Snapshot()` in runtime loop | Router resolves functions only in statically-configured namespaces |
| `pkg/buildermgr/buildermgr.go` | `StartNSWatcher(...)` call added | NSWatcher never starts in buildermgr |
| `pkg/buildermgr/envwatcher.go` | `AddNamespace(ns string)` — starts per-NS Environment informer | Buildermgr ignores Environments in new namespaces |
| `pkg/buildermgr/pkgwatcher.go` | `AddNamespace(ns string)` — starts per-NS Package informer | Buildermgr ignores Packages in new namespaces |
| `pkg/storagesvc/archivePruner.go` | Uses `Snapshot()` instead of static namespace list | Archive pruner only cleans old namespaces |
| `.gitignore` | Added `*.token` | Minor — token files would be committed accidentally |
| `deploy/multitenant/rbac.yaml` | New file (see above) | — |

---

## 5. Data Flow: from label to HTTP 200

```
kubectl label ns tenant-abc123 fission.io/managed=true
        │
        ▼
k8s API server emits ADDED event on Namespace stream
        │
        ▼ (within ~50ms)
utils.RunManagedNamespaceWatcher  ← informer AddFunc
        │  (all 3 components share this function)
        ▼
NamespaceManager.DispatchAdd(ctx, "tenant-abc123")
        │
        ├──► executor subscriber:
        │        registerNamespace()
        │          1. DefaultNSResolver().AddNamespace("tenant-abc123")
        │          2. EnsureNamespaceSA(ctx, client, logger, "tenant-abc123")
        │               └─ creates fission-fetcher SA + Role + RoleBinding
        │          3. poolmgr.AddNamespace("tenant-abc123")
        │               └─ creates per-NS InformerFactory, pod lister, handlers
        │          4. newdeploy.AddNamespace("tenant-abc123")
        │          5. container.AddNamespace("tenant-abc123")
        │
        ├──► router subscriber:
        │        HTTPTriggerSet.AddNamespace("tenant-abc123")
        │          └─ starts watching HTTPTriggers + Functions in that NS
        │
        └──► buildermgr subscriber:
                 envWatcher.AddNamespace("tenant-abc123")
                 pkgWatcher.AddNamespace("tenant-abc123")
                   └─ starts watching Environments + Packages in that NS

User creates: fission env create --namespace tenant-abc123 ...
              fission fn create   --namespace tenant-abc123 ...
              fission httptrigger create --namespace tenant-abc123 ...

Router picks up HTTPTrigger → resolves Function → cold-starts pod in tenant-abc123
        │
        ▼
HTTP 200 ← function response
```

---

## 6. RBAC Explained

Three ClusterRoles in `deploy/multitenant/rbac.yaml`:

### `fission-executor-ns-watcher` (also for router: `fission-router-ns-watcher`)
```
namespaces: list, watch
```
Without this: informer fails to start with "Forbidden" — components never learn
about new namespaces.

### `fission-executor-sa-provisioner`
```
serviceaccounts:             get, list, watch, create, update, patch
events:                      create
localsubjectaccessreviews:   create
roles:                       get, list, watch, create, update, patch
rolebindings:                get, list, watch, create, update, patch
```
Why `events.create`: Kubernetes forbids creating a Role that grants permissions
the caller does not currently hold. Since `fission-fetcher` gets `events.create`
in the Role we create for it, `fission-executor` must hold `events.create` itself
to be allowed to create that Role. This is a k8s RBAC escalation prevention rule.

Why `localsubjectaccessreviews.create`: `EnsureNamespaceSA` calls
`setupSAAndRoleBindings` which first checks if the SA already has each permission
before creating it — that check uses a `LocalSubjectAccessReview`.

---

## 7. Key Design Decisions and Why

### Decision: pub/sub via `NamespaceManager`, not direct calls
**Why not:** simply call `poolmgr.AddNamespace`, `router.AddNamespace` etc.
directly from a shared goroutine.
**Why pub/sub:** executor, router and buildermgr run as separate processes
(different pod). Each process has its own copy of the watcher. Subscribers are
registered in-process. This pattern makes each component fully self-contained and
testable in isolation. No cross-process coupling.

### Decision: `fission.io/managed=true` label as the only trigger
**Why not:** watch all namespaces, or use a CRD, or use annotations.
**Why label:** labels are the idiomatic k8s way to select resources. A label
selector in the informer factory (`fission.io/managed=true`) means the informer
only receives events for labeled namespaces — zero overhead for the hundreds of
system namespaces.

### Decision: `EnsureNamespaceSA` in executor, not in a separate operator
**Why:** the SA must exist before the first pool pod starts. The executor is
already in the hot path — it processes the NS event and then immediately triggers
pod scheduling. Doing it in a separate controller would introduce a race. Doing it
in the executor keeps the lifecycle coupled correctly.

### Decision: `NamespaceRemovalStrategy = track-only` (not `dispatch-remove`)
**Why:** removing a namespace in k8s is already an irreversible event — all
resources inside are cascade-deleted by k8s. The executor types detect the pod
deletions themselves. Dispatching an explicit "remove" to all subscribers would
require each subscriber to implement a cleanup path — added complexity for zero
operational benefit in our use case. Can be switched per-component via
`ManagedNamespaceWatcherConfig.RemovalStrategy`.

### Decision: `AddNamespace` is idempotent (safe to call multiple times)
**Why:** k8s informers can deliver the same event more than once (resync). Every
`AddNamespace` call is a no-op if the namespace is already registered. No locks
held across the whole function — `NamespaceResolver.AddNamespace` uses a write
lock only for the map write, checks for existence first under the same lock.

### Decision: global `NamespaceResolver` (`DefaultNSResolver()`) updated once in executor
**Why:** `DefaultNSResolver().AddNamespace(ns)` is called exactly once in
`registerNamespace()` — before the executor types are called. Each executor type
does NOT call it themselves. This avoids a subtle race: if executor type A adds
the NS to the global resolver first, and executor type B checks the global resolver
as its dedup mechanism, B would see it as already registered and skip — even
though B has not actually processed it yet. The correct dedup is per executor type.

---

## 8. What Is NOT Done (deliberately deferred)

| Missing feature | Why deferred | File/interface to extend |
|----------------|--------------|--------------------------|
| `dispatch-remove` full wiring | Not needed for current use case | `ManagedNamespaceWatcherConfig.RemovalStrategy` — just switch the constant |
| Buildermgr ClusterRole in rbac.yaml | Buildermgr uses the same SA as executor in our deployment; check your setup | Add a third ClusterRole/Binding to `deploy/multitenant/rbac.yaml` |
| Helm chart integration | We apply rbac.yaml manually. A proper Helm chart would include these RBAC objects | `charts/fission-all/templates/` |
| Layer 2 (tenant isolation: per-NS network policy, resource quotas) | Out of scope for Layer 1 | Not started |
| Layer 3 (per-tenant auth, billing hooks) | Out of scope | Not started |
| `NamespaceManager.DispatchRemove` subscriber wiring | Each subscriber has a `RemoveFunc` stub returning nil | Implement per subscriber |

---

## 9. Porting to a New Upstream Version

When `github.com/fission/fission` releases v1.23 or later, follow this order:

1. **Check the upstream changelog** for any changes to:
   - `pkg/utils/namespace.go` — if they renamed or refactored `NamespaceResolver`, our `AddNamespace`/`Snapshot` additions need to be reapplied
   - `pkg/executor/executortype/executortype.go` — if they changed the `ExecutorType` interface, our `AddNamespace` method needs to be reapplied
   - `pkg/router/httpTriggers.go` — if `HTTPTriggerSet` changed, our `AddNamespace` method on it needs to be reapplied
   - `pkg/buildermgr/envwatcher.go`, `pkgwatcher.go` — same
   - `pkg/utils/informer.go` — if the informer factory pattern changed

2. **Apply in this order** (each depends on the previous):
   1. `pkg/utils/namespace_manager_model.go` — pure types, no deps on other changed files
   2. `pkg/utils/namespace.go` additions (`AddNamespace`, `Snapshot`, label constants)
   3. `pkg/utils/namespace_manager.go` — depends on model + namespace.go
   4. `pkg/utils/serviceaccount.go` — add `EnsureNamespaceSA` at the bottom
   5. `pkg/utils/informer.go` — add `NewSharedInformerFactoryForNamespaces`
   6. `pkg/executor/executortype/executortype.go` — add `AddNamespace` to interface
   7. Executor types: `poolmgr/gpm.go`, `newdeploy/newdeploymgr.go`, `container/containermgr.go` — implement `AddNamespace`
   8. `pkg/executor/multitenant/` — copy the whole package as-is
   9. `pkg/executor/executor.go` — add `StartNSWatcher` call
   10. `pkg/router/httpTriggers.go` — add `AddNamespace` method on `HTTPTriggerSet`
   11. `pkg/router/namespace_subscriber.go`, `pkg/router/ns_watcher.go` — copy as-is
   12. `pkg/router/router.go` — add `StartNSWatcher` call
   13. `pkg/buildermgr/envwatcher.go`, `pkgwatcher.go` — add `AddNamespace` method
   14. `pkg/buildermgr/namespace_subscriber.go`, `pkg/buildermgr/ns_watcher.go` — copy as-is
   15. `pkg/buildermgr/buildermgr.go` — add `StartNSWatcher` call
   16. `pkg/storagesvc/archivePruner.go` — replace static namespace list with `Snapshot()`
   17. `deploy/multitenant/rbac.yaml` — apply unchanged

3. **Run tests:**
   ```bash
   go test ./pkg/utils/... ./pkg/executor/... ./pkg/router/... ./pkg/buildermgr/...
   ```

4. **Build and deploy:**
   ```bash
   # on VM:
   docker run --rm -v $PWD:/src -w /src golang:1.26-alpine \
     sh -c 'go build -o /src/fission-bundle-bin ./cmd/fission-bundle/'
   docker build -f Dockerfile.mt -t naeel/fission-bundle:vNEW_TAG .
   docker push naeel/fission-bundle:vNEW_TAG
   kubectl set image deployment/executor -n fission executor=naeel/fission-bundle:vNEW_TAG
   kubectl set image deployment/router   -n fission router=naeel/fission-bundle:vNEW_TAG
   kubectl set image deployment/buildermgr -n fission buildermgr=naeel/fission-bundle:vNEW_TAG
   ```

5. **Run e2e test:**
   ```bash
   bash ~/terra/fission/scripts/test_layer1.sh
   # Expected: PASS=5 FAIL=0
   ```

---

## 10. Test Coverage

| Test file | What it covers |
|-----------|---------------|
| `pkg/utils/namespace_test.go` | `AddNamespace` dedup, `Snapshot` sorted output, `IsManagedNamespace` |
| `pkg/utils/namespace_manager_test.go` | `Bootstrap`, `DispatchAdd`/`Remove`/`Resync`, subscriber dispatch, `TestStartManagedNamespaceWatcherIntegration` — full k8s fake informer → subscriber pipeline |
| `pkg/utils/namespace_manager_model_test.go` | Model helpers, `Clone`, `IsActive`, `IsTerminal` |
| `pkg/executor/multitenant/ns_watcher_test.go` | `registerNamespace` with fake k8s client |
| `pkg/executor/multitenant/namespace_subscriber_test.go` | Subscriber adapter |
| `pkg/router/namespace_subscriber_test.go` | Router subscriber adapter |
| `pkg/buildermgr/namespace_subscriber_test.go` | Buildermgr subscriber adapter |
| `~/terra/fission/scripts/test_layer1.sh` | End-to-end: create labeled NS → executor registers it → create env/fn/trigger → call function → HTTP 200 |