Nail/SQS-service
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23 Commits 6 Branches 0 Tags
a1ff9c4e52ddee2ce10a7fd89bbc8150133359ac
Commit Graph
23 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Naeel a1ff9c4e52 docs: finalize 64kb ingress investigation 2026-04-12 07:59:04 +03:00
Naeel eba01c9580 fix: 7 performance/correctness fixes — v0.1.20 
1. Убрано логирование тела сообщения (256KB I/O на каждый send — perf+security)
2. SentTimestamp исправлен: m.SentTime вместо time.Now() (баг)
3. MD5 не пересчитывается на ReceiveMessage — используется кэш из SendMessage
4. ChangeMessageVisibility/batch теперь персистит в Redis (баг — потеря данных)
5. MessageDoesNotExist error code: QueueExists → ReceiptHandleIsInvalid (copy-paste баг)
6. copystructure убран из GetQueueAttributes — простой map lookup
7. SNS dead code удалён (SnsErrors, SnsErrorType — не используется в SQS сервисе)

Tested: quick_test 31/31 PASS, deployed v0.1.20
 2026-04-11 20:39:08 +03:00
Naeel c580101f0c doc: анализ деградации на больших payload (3 root causes) 
- json.Marshal(queue) сериализует ВСЕ сообщения под Lock — O(N*msg_size)
- log.Infof логирует полное тело каждого сообщения (256KB на запись)
- CPU limit 500m вызывает K8s CFS throttling на тяжёлых операциях
- PeriodicTasks держит глобальный Lock каждую секунду
- Баг: MessageDoesNotExist возвращает Code=QueueExists
 2026-04-11 20:19:28 +03:00
Naeel 406ec69679 feat: add full comparative benchmark + stress test v2 
- benchmark_full.sh: 10-section comparative benchmark (shared-sqs vs Yandex MQ)
  All 17 SQS commands tested, latency, throughput, message sizes, batch, long poll
- stress_test_v2.sh: improved stress test
- Key findings: large payload (64KB+) degradation needs investigation
 2026-04-11 20:13:00 +03:00
Naeel b181b7ef0b test: compare_sqs.sh benchmark Yandex MQ vs shared-sqs + thinking log 2026-04-11 19:46:01 +03:00
Naeel f618f02e76 doc: stress testing report, progress update, thinking log — v0.1.19 assessment 2026-04-11 19:15:57 +03:00
Naeel eaed7bd6c2 test: уменьшаю нагрузку для SSH-стабильности (10msg/worker, 30/tenant, longpoll 20s) 2026-04-11 17:03:41 +03:00
Naeel 24103310c2 test: stress_test v2 — снижение параллелизма до 20 воркеров (SSH stability) 2026-04-11 16:52:31 +03:00
Naeel bd8303ca2e test: stress_test.sh v2 — 50 воркеров, 1000 msg, long-poll, race, batch, 30 очередей, memory, multi-kill, verbose progress 2026-04-11 16:41:07 +03:00
Naeel f937b7fccb fix: stress_test.sh — правильный формат queue URL, парсинг tenant API, subshell-safe массивы 2026-04-11 16:28:59 +03:00
Naeel 60931fd2fd test: stress_test.sh — конкурентность, multi-tenant изоляция, burst, pod kill, Redis disconnect 2026-04-11 16:21:08 +03:00
Naeel d633e59a50 fix: v0.1.19 — валидация VisibilityTimeout/WaitTimeSeconds/пустой MessageBody, fix hardcore_test LONG_Q length 2026-04-11 15:56:31 +03:00
Naeel 95d1dc1adb chore: удалить временный диагностический скрипт 2026-04-11 09:44:53 +03:00
Naeel 38194fb2fa feat: v0.1.18 — TagQueue/UntagQueue/ListQueueTags/ChangeMessageVisibilityBatch, JWT auth in UI API tests, hardcore_test JWT fix, Helm chart v0.1.18 2026-04-11 09:44:38 +03:00
Naeel fbecde72eb feat: add 4 missing SQS API commands for Yandex/AWS compatibility 
- ChangeMessageVisibilityBatch: batch visibility timeout (up to 10 msgs)
- TagQueue: add/update queue tags
- UntagQueue: remove queue tags by keys
- ListQueueTags: list all queue tags
- Added Tags field to Queue struct
- Request/Response models for all 4 commands
- Registered in router (17 total API commands now)
- Yandex Message Queue API reference doc
 2026-04-11 07:51:17 +03:00
Naeel c9d9231efc docs: record compatibility runs, decisions and session reasoning 2026-04-10 20:27:59 +03:00
Naeel ebcb20475a perf: optimize receive long polling and finalize formatting cleanup 2026-04-10 19:49:55 +03:00
Naeel 3b4fe0bb3a security: address medium risks in jwt, redis ordering and body limit 2026-04-10 19:44:34 +03:00
Naeel a5e9bfb15c security: fix critical/high auth, idor, races and persistence 2026-04-10 19:41:59 +03:00
Naeel e9a26f7975 v0.1.17: security hardening — 18/20 vulnerabilities fixed 
Phase 1 (Critical):
- #1 JWT auth (done in v0.1.16)
- #2 Batch message size validation in send_message_batch.go
- #10 RLock in GetQueueUrlV1 (data race fix)

Phase 2 (AWS-compatible limits):
- #3 QueueName validation: max 80 chars, [a-zA-Z0-9_-](.fifo)?
- #4 WaitTimeSeconds clamped to 0-20
- #5 ReceiveMessageWaitTimeSeconds clamped to 0-20
- #6 DelaySeconds clamped to 0-900
- #7 VisibilityTimeout clamped to 0-43200
- #8 MaxNumberOfMessages clamped to 1-10
- #9 Message attributes limited to 10 per message
- #15 BatchEntryId length validated (max 80)
- #16 DeduplicationID length validated (max 128)
- #17 GroupID length validated (max 128)

Phase 3 (Per-tenant resource limits):
- #11 Max messages per queue (120K standard, 20K FIFO)
- #12 Global tenant limit (1000)
- #3.5 HTTP request body size limit (1MB via MaxBytesReader)

Phase 4 (Stability):
- #14 Duplicates map cleanup (already in PeriodicTasks)
- #13 FIFO group lock timeout (already in visibility timeout reset)
- #18 Redis size guard: skip save if >50MB

Skipped (Low, no real risk):
- #19 {account} URL param (informational only, not used for access)
- #20 ReceiptHandle format (self-validating UUID#UUID)

New file: app/gosqs/validation.go — centralized AWS SQS limits and validators
 2026-04-10 18:58:40 +03:00
Naeel b9d434bcc5 v0.1.16: JWT auth via nubes API, auto-provisioning, UI login 
- app/auth/jwt.go: ParseJWTClaims, TenantIDFromSub (sless-compatible SHA256), PingNubesAPI
- app/admin/admin.go: POST /ui/api/auth endpoint, jwtMiddleware for /ui/api/*
- app/tenant/tenant_store.go: NubesSub/Email fields, GetBySub, CreateFromJWT
- app/ui/index.html: login page, email in navbar, JWT session in localStorage
- deployments/k8s/deployment.yaml: v0.1.16, NUBES_ENDPOINT env
- doc/decisions/resource-limits-plan.md: 20 vulnerabilities audit
- Fix: /ui/api/auth moved to subrouter (gorilla/mux PathPrefix conflict)
 2026-04-10 18:35:19 +03:00
Naeel 6fb160f8ae docs: add ONBOARDING.md — full context memo for new chat sessions 2026-04-10 16:51:55 +03:00
Naeel c3ba2dcae4 chore: initial import from sless/shared-sqs (v0.1.14) 
- Standalone SQS-service repository
- Multi-tenant message queue service, AWS SQS compatible
- Based on GoAws, with mutable tenants, auth, WebUI, Redis persistence
- Ready for independent development and deployment
- See doc/ and README.md for architecture and usage
 2026-04-10 16:47:27 +03:00