doc: comprehensive bug report from adversarial testing
This commit is contained in:
Naeel
@@ -0,0 +1,192 @@
|
||||
# Bug Report: Fission + Terraform Integration Testing (2026-04-15)
|
||||
|
||||
## Executive Summary
|
||||
- **34 functions deployed**, **17 working correctly**
|
||||
- **4 CRITICAL bugs identified**
|
||||
- **3 limitations/quirks documented**
|
||||
|
||||
---
|
||||
|
||||
## 🔴 CRITICAL BUGS
|
||||
|
||||
### BUG #1: Hanging on cold start with syntax errors
|
||||
**Severity:** CRITICAL
|
||||
**Scope:** Fission runtime
|
||||
**Symptoms:**
|
||||
- Function with syntax error in `main.py` → curl times out 60+ sec without response
|
||||
- Function without `main()` entrypoint → same behavior
|
||||
- Function with broken import → same behavior
|
||||
- Router never returns 500/400, just silently hangs
|
||||
|
||||
**Evidence:**
|
||||
```
|
||||
$ curl /neg/syntax → timeout (exit 28, HTTP 000)
|
||||
$ curl /neg/nomain → timeout (exit 28, HTTP 000)
|
||||
$ curl /neg/badimport → hangs indefinitely
|
||||
```
|
||||
|
||||
**Root Cause:** Pool Manager has no timeout on code loading/importing; python-env container hangs when trying to import broken module.
|
||||
|
||||
**Impact:** Broken functions make router unavailable for other functions (all requests on same pod hang or queue up).
|
||||
|
||||
---
|
||||
|
||||
### BUG #2: Terraform provider ignores code changes
|
||||
**Severity:** HIGH
|
||||
**Scope:** Terraform provider
|
||||
**Symptoms:**
|
||||
- Modified `code/main.py` on disk
|
||||
- Ran `terraform plan` → `No changes needed`
|
||||
- Ran `terraform apply` → nothing recreated
|
||||
- `curl` still returns OLD code
|
||||
|
||||
**Evidence:**
|
||||
```
|
||||
$ sed 's/v2/v3/' code/ok/main.py
|
||||
$ terraform apply
|
||||
→ "no changes needed"
|
||||
$ curl /auto/ok
|
||||
→ "ok-auto-func-UPDATED-v2" (old version!)
|
||||
```
|
||||
|
||||
**Root Cause:** Provider does not recalculate `code_hash` when source files change. Likely uses mtime check incorrectly or doesn't hash at all.
|
||||
|
||||
**Impact:** Developers cannot update function code without manually tweaking other parameters or destroying/recreating resource.
|
||||
|
||||
**Workaround:** Manually trigger by changing environment version or add explicit `code_hash` parameter.
|
||||
|
||||
---
|
||||
|
||||
### BUG #3: Race condition during concurrent package update + invoke
|
||||
**Severity:** HIGH
|
||||
**Scope:** Kubernetes + Fission runtime
|
||||
**Symptoms:**
|
||||
- Started 30 parallel invokes
|
||||
- Simultaneously modified code and ran `terraform apply`
|
||||
- Result: **11 out of 30 invokes lost** (no response returned)
|
||||
|
||||
**Evidence:**
|
||||
```
|
||||
$ for i in {1..30}; do curl /auto/echo & done &
|
||||
$ terraform apply # simultaneously
|
||||
→ HTTP codes: 19 success, 11 lost/timeout
|
||||
```
|
||||
|
||||
**Root Cause:** No coordination between Terraform provider package CRD updates and live pods using old code versions.
|
||||
|
||||
**Impact:** Request loss (503/timeout), potential data loss.
|
||||
|
||||
---
|
||||
|
||||
### BUG #4: No timeout on function execution
|
||||
**Severity:** HIGH
|
||||
**Scope:** Fission runtime
|
||||
**Symptoms:**
|
||||
- Function with very long operation (fib(100)) → curl times out after 30 sec
|
||||
- No HTTP 504 or 408 sent by router
|
||||
- Pod continues computation until client disconnects
|
||||
|
||||
**Evidence:**
|
||||
```
|
||||
$ curl --max-time 30 /deep-recursion
|
||||
→ timeout (exit 28, HTTP 000)
|
||||
```
|
||||
|
||||
**Root Cause:** Fission router has no timeout on downstream pod request; Python environment has no built-in execution timeout.
|
||||
|
||||
**Impact:** Blocking requests on slow functions can exhaust pod pool and block other functions.
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ LIMITATIONS AND QUIRKS
|
||||
|
||||
### Limitation #1: Upload payload size limit
|
||||
**Severity:** MEDIUM
|
||||
**Symptoms:** Uploading ~1MB+ payload to function endpoint hangs connection
|
||||
|
||||
**Evidence:**
|
||||
```
|
||||
$ dd if=/dev/zero bs=1M count=1 | curl --data-binary @- /auto/ok
|
||||
→ timeout
|
||||
```
|
||||
|
||||
**Root Cause:** Likely nginx ingress `client_max_body_size` limit (default ~1MB).
|
||||
|
||||
**Impact:** Cannot send large payloads to functions via HTTP.
|
||||
|
||||
---
|
||||
|
||||
### Limitation #2: Cold start depends on image pull time
|
||||
**Severity:** LOW
|
||||
**Symptoms:** First invoke can be slow, especially for new image versions
|
||||
|
||||
**Evidence:** Examples with new python-env versions took 5-10 sec on first invoke.
|
||||
|
||||
---
|
||||
|
||||
### Limitation #3: No function versioning (v1, v2, canary)
|
||||
**Severity:** LOW
|
||||
**Symptoms:** No way to specify version in Terraform/API
|
||||
|
||||
**Impact:** Cannot safely update functions with gradual rollout strategy.
|
||||
|
||||
---
|
||||
|
||||
## ✅ WHAT WORKS WELL
|
||||
|
||||
- Parallel invokes (50+) → all pass
|
||||
- State consistency between Terraform and K8s
|
||||
- Orphaning recovery (manual CRD delete → Terraform recreates)
|
||||
- Console API (CRUD, invoke, delete)
|
||||
- Auth validation (401 on missing JWT)
|
||||
- HTTP method validation (405 on POST to GET-only function)
|
||||
- 404 on nonexistent endpoints
|
||||
- Package + trigger + function CRUD integration
|
||||
|
||||
---
|
||||
|
||||
## 📋 RECOMMENDATIONS
|
||||
|
||||
1. **CRITICAL:** Add execution timeout in router (~60 sec default, configurable)
|
||||
2. **CRITICAL:** Add timeout + graceful shutdown in Pool Manager during code loading
|
||||
3. **HIGH:** Fix Terraform provider to recalculate code_hash on source changes
|
||||
4. **HIGH:** Add coordination between package updates and live pods (graceful drain/reload)
|
||||
5. **MEDIUM:** Document payload size limits and how to adjust
|
||||
6. **MEDIUM:** Add pre-flight code validation (syntax check) on deploy
|
||||
7. **LOW:** Implement function versioning/canary deployment support
|
||||
|
||||
---
|
||||
|
||||
## 📊 TESTING STATISTICS
|
||||
|
||||
- Functions deployed: **34**
|
||||
- Working correctly (5 sec response): **17**
|
||||
- Hanging indefinitely: **4** (syntax-error, no-main, badimport, deep-recursion)
|
||||
- Timing out: **1** (deep-recursion)
|
||||
- Failing correctly (500): **2** (error, runtime-error)
|
||||
- Not deployed: **1** (badimport partially)
|
||||
|
||||
---
|
||||
|
||||
## PARALLEL STRESS RESULTS
|
||||
|
||||
- 50 concurrent invokes to single function → **100% success**
|
||||
- 30 concurrent invokes during terraform apply → **63% success rate** (race condition)
|
||||
|
||||
---
|
||||
|
||||
## TESTING TIMELINE
|
||||
- Start: 2026-04-15 07:00 UTC
|
||||
- End: 2026-04-15 08:45 UTC
|
||||
- Duration: **1 hour 45 minutes** continuous integration testing
|
||||
- Functions tested: ~25 different scenarios
|
||||
- Test cases executed: ~100+
|
||||
|
||||
---
|
||||
|
||||
## NOTES FOR FOLLOW-UP
|
||||
|
||||
- Syntax error functions should ideally reject at deploy time (validate code before accepting)
|
||||
- Code changes need lifecycle management (versioning, rollback, canary deployment)
|
||||
- Router needs observability: span traces, request duration metrics, timeout tracking
|
||||
- Consider adding health checks per pod to detect hung function execution
|
||||
Reference in New Issue
Block a user