Problem
-------
The namespace reconciler (RunReconciler, added previously) retries namespaces
in NamespacePhaseFailed every 30s by calling DispatchResync. But the phase
could never actually reach NamespacePhaseFailed for the executor component
because the executor's NamespaceSubscriber always returned nil — swallowing
any SA-provisioning or informer-init errors. The reconciler was dead code for
the executor path.
Root cause chain
----------------
1. setupSAAndRoleBindings() — void, errors only logged internally.
2. EnsureNamespaceSA() — void, just called setupSAAndRoleBindings.
3. registerNamespace() — void, errors from both functions lost.
4. Executor AddFunc/ResyncFunc — always returned nil to dispatch().
5. dispatch() marks parts Active unconditionally → NamespacePhaseFailed
is never triggered for executor → RunReconciler never fires for executor.
Consequence: if EnsureNamespaceSA failed (transient k8s 503, RBAC webhook
timeout, etc.) the namespace appeared Active in the manager but the fetcher
ServiceAccount was missing. Pool pods would CrashLoopBackOff on every call
to that namespace until a full process restart.
Changes
-------
pkg/utils/serviceaccount.go
- setupSAAndRoleBindings: void → error. Returns the first k8s API error
so callers can decide whether to retry.
- runSACheck: ignores the error with _ = (same behaviour as before, it's
a periodic background loop that already logs internally).
- EnsureNamespaceSA: void → error, propagates setupSAAndRoleBindings.
Updated godoc to explain the retry contract.
pkg/executor/multitenant/ns_watcher.go
- registerNamespace: void → error.
* EnsureNamespaceSA error → wrapped as 'EnsureNamespaceSA: ...' and returned.
* registerExecutorTypes error → wrapped as 'registerExecutorTypes: ...' and returned.
* Success log line only emitted when both succeed.
- Added 'fmt' import for error wrapping.
pkg/executor/multitenant/namespace_subscriber.go
- AddFunc: return registerNamespace(...) instead of ignoring its error.
- ResyncFunc: same — plus a comment explaining why it is safe to call
registerNamespace again (SA creation is idempotent, executor-type
AddNamespace guards against duplicate informer creation).
pkg/utils/namespace_manager.go
- RunReconciler interface signature: added *zap.Logger parameter.
Callers pass the component logger so retries are visible in prod logs.
- RunReconciler implementation:
* Accepts logger; falls back to zap.NewNop() if nil.
* Skips the tick entirely when no failed namespaces are found (no log spam).
* Logs 'retrying failed namespaces' with count + list when found.
* Logs per-namespace 'dispatching resync'.
* Logs 'resync succeeded' or 'resync still failing, will retry' with error.
- RunManagedNamespaceWatcher: passes logger to RunReconciler.
End-to-end flow after this fix
-------------------------------
1. EnsureNamespaceSA fails (k8s 503).
2. registerNamespace returns error.
3. Executor AddFunc returns error.
4. dispatch() calls MarkPartFailed("executor") → deriveNamespacePhase →
NamespacePhaseFailed.
5. RunReconciler tick (30s) finds the namespace → DispatchResync →
registerNamespace called again → EnsureNamespaceSA (idempotent) →
if API recovered: success → MarkPartActive → NamespacePhaseActive.
6. Log line 'namespace reconciler: resync succeeded' confirms recovery.
Backward compatibility
----------------------
- NamespaceManager interface: RunReconciler gained a *zap.Logger param.
There is exactly one implementation (inMemoryNamespaceManager) and one
call site (RunManagedNamespaceWatcher). No external mocks.
- EnsureNamespaceSA: callers outside this codebase (if any) that ignore
the error will still compile (Go allows ignoring return values).
- All 26 affected tests pass: go test ./pkg/utils/... ./pkg/executor/...
./pkg/buildermgr/... ./pkg/router/...
- DefaultNSResolver.RemoveNamespace(): removes NS from global map on label removal
so Snapshot() and idleObjectReaper stop iterating deleted namespaces.
Fixes class of dirty-state bugs when NS name is reused by new tenant.
- HandleWatcherNamespaceRemoval: call RemoveNamespace on both TrackOnly and
DispatchRemove strategies — global resolver cleanup is always required.
- dispatch(): parallel subscriber execution via goroutine per subscriber +
sync.WaitGroup. Reduces onboarding latency from O(N_subscribers × API_latency)
to O(max(API_latency)). Safe: MarkPart* are internally mutex-protected.
- inMemoryNamespaceManager.RunReconciler(): 30s ticker scans for
NamespacePhaseFailed records and retries via DispatchResync. Started
automatically by RunManagedNamespaceWatcher. Fixes permanent stuck-failed
state caused by transient k8s API errors.
Analysis source: FORENSIC_ARCHITECTURE_AUDIT.md §Deep Risk Analysis
“Naeel”