Nail/SQS-service
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
138 Commits 6 Branches 0 Tags
ecf66e4b27b930e72bd934e2c9610f030c3f144b
Commit Graph
37 Commits
Author SHA1 Message Date
“Naeel” 7467eccd66 v0.1.35: фиксы — синхронные удаления в Redis (воскрешение очередей) + RedrivePolicy tenant-scoped DLQ 2026-08-15 08:57:25 +04:00
“Naeel” af0325ebbe v0.1.35: серверные таймауты по Соннету — ReadHeaderTimeout вместо ReadTimeout, MaxHeaderBytes 2026-08-15 08:34:02 +04:00
“Naeel” c4da02c814 v0.1.35: устранены гонки map (ChangeVisibility/DeleteBatch) и nil-deref под Lock + OverLimit 2026-08-14 18:48:35 +04:00
“Naeel” 0b56677363 v0.1.35: лимит очередей JWT-тенантов 50, обновление лимита при логине 2026-08-14 16:51:12 +04:00
“Naeel” fa237204f0 v0.1.35: FifoQueue/ContentBasedDeduplication в whitelist (совместимость CLI v2) 2026-08-14 16:02:51 +04:00
“Naeel” 7f6e9308d5 v0.1.35: whitelist имён атрибутов и для JSON-протокола (UnmarshalJSON) 2026-08-14 16:00:48 +04:00
“Naeel” dbb1aea936 v0.1.35: resolveProtocol — query-mode клиенты парсят JSON-ответы (эмпирика), ошибки AWS-JSON __type 2026-08-14 15:56:40 +04:00
“Naeel” ec10acc99e v0.1.35: query-mode XML-ответы, TOCTOU receive, валидации AWS, purge 60s, сужение локов 2026-08-14 15:53:08 +04:00
“Naeel” 0f24658fc7 v0.1.34: кэш валидации JWT (TTL 5 мин + grace 2 мин) — консоль устойчива к флаку шлюза 2026-08-14 14:09:49 +04:00
“Naeel” 12d07f1589 v0.1.33: строгий нейтральный стиль текстов на главной и в примерах 2026-08-14 13:49:26 +04:00
“Naeel” 94108eec48 v0.1.32: главная — пошаговый старт; примеры команд — на /examples крупным шрифтом; консоль сразу в аккаунт 2026-08-14 13:45:06 +04:00
“Naeel” afa9a1a7e4 v0.1.31: админский UI /admin, юзерский UI без слова «тенант» 2026-08-14 10:19:35 +04:00
“Naeel” bbbd17ad45 v0.1.30: локальные логотип и favicon /static в консоли (внешние terra.k8c.ru битые) 2026-08-14 10:10:18 +04:00
“Naeel” cfea73cdea v0.1.29: светлый Nubes-дизайн консоли, читаемая модалка Credentials (env + файл кредов) 2026-08-14 09:56:18 +04:00
“Naeel” e19f63abe4 v0.1.28: email-идентичность — детерминированные креды из email, credentials-эндпоинт, UI модалка 2026-08-14 09:43:36 +04:00
“Naeel” af656fe8f5 v0.1.27: страница-описание сервиса на корне / (дизайн Nubes) + /static 2026-08-14 09:05:16 +04:00
“Naeel” 3a105de05b v0.1.26: версия в /health, admin health и UI; ldflags VERSION; Makefile build фикс 2026-08-14 08:19:14 +04:00
“Naeel” 166baa9a98 fix: валидация JWT перебором всех стендов gateway (токен любого стенда принимается) 2026-08-14 08:05:16 +04:00
“Naeel” de72fe5c8f fix: JWT-валидация через lk-api-gateway PROD + User-Agent в PingNubesAPI 2026-08-14 07:51:05 +04:00
“Naeel” 032ca66cc6 feat: UI — инфо-блок о сервисе на странице входа 2026-08-13 23:20:51 +04:00
“Naeel” 15ec1a9ee4 chore: удалить устаревшие тестовые файлы (SNS-легаси, несуществующие импорты) 2026-08-13 22:06:26 +04:00
Naeel a4c910a099 feat(metrics): integrate Prometheus metrics into router and main 
- /metrics endpoint via promhttp.Handler
- actionHandler: request counter, bytes counter, latency histogram, errors
- Gauge updater: queues/messages count per tenant every 15s
- All metrics labeled by tenant and/or operation
 2026-04-12 12:04:54 +03:00
Naeel 3d59b5500c feat(metrics): add Prometheus metrics package for Victoria Metrics 
Metrics exposed:
- sqs_requests_total{tenant, operation} — request counter
- sqs_request_bytes_total{tenant, operation} — traffic counter
- sqs_request_duration_seconds{operation} — latency histogram
- sqs_errors_total{operation} — error counter
- sqs_queues_count{tenant} — gauge, updated every 15s
- sqs_messages_count{tenant} — gauge, updated every 15s
 2026-04-12 12:03:16 +03:00
Naeel aade320400 feat(billing): add queue_name column to usage records 2026-04-12 11:56:53 +03:00
Naeel 1272388673 feat: billing — usage tracking per SQS operation in PostgreSQL 
- New package app/billing: Init/RecordUsage/Close with auto-migrate
- Integration in actionHandler: record tenant_id, operation, msg_count, msg_bytes
- Helm chart: billing section in values.yaml, secret-billing.yaml, env vars in deployment
- Optional: billing disabled by default (BILLING_PG_HOST not set = no-op)
- Table: sqs_usage_records with index on (tenant_id, recorded_at)
 2026-04-12 11:51:27 +03:00
Naeel d7e9d90453 Add demo UI showcase mode 2026-04-12 09:58:32 +03:00
Naeel 4a83575491 feat: persist messages separately and update ingress docs 2026-04-12 08:44:46 +03:00
Naeel eba01c9580 fix: 7 performance/correctness fixes — v0.1.20 
1. Убрано логирование тела сообщения (256KB I/O на каждый send — perf+security)
2. SentTimestamp исправлен: m.SentTime вместо time.Now() (баг)
3. MD5 не пересчитывается на ReceiveMessage — используется кэш из SendMessage
4. ChangeMessageVisibility/batch теперь персистит в Redis (баг — потеря данных)
5. MessageDoesNotExist error code: QueueExists → ReceiptHandleIsInvalid (copy-paste баг)
6. copystructure убран из GetQueueAttributes — простой map lookup
7. SNS dead code удалён (SnsErrors, SnsErrorType — не используется в SQS сервисе)

Tested: quick_test 31/31 PASS, deployed v0.1.20
 2026-04-11 20:39:08 +03:00
Naeel d633e59a50 fix: v0.1.19 — валидация VisibilityTimeout/WaitTimeSeconds/пустой MessageBody, fix hardcore_test LONG_Q length 2026-04-11 15:56:31 +03:00
Naeel 38194fb2fa feat: v0.1.18 — TagQueue/UntagQueue/ListQueueTags/ChangeMessageVisibilityBatch, JWT auth in UI API tests, hardcore_test JWT fix, Helm chart v0.1.18 2026-04-11 09:44:38 +03:00
Naeel fbecde72eb feat: add 4 missing SQS API commands for Yandex/AWS compatibility 
- ChangeMessageVisibilityBatch: batch visibility timeout (up to 10 msgs)
- TagQueue: add/update queue tags
- UntagQueue: remove queue tags by keys
- ListQueueTags: list all queue tags
- Added Tags field to Queue struct
- Request/Response models for all 4 commands
- Registered in router (17 total API commands now)
- Yandex Message Queue API reference doc
 2026-04-11 07:51:17 +03:00
Naeel ebcb20475a perf: optimize receive long polling and finalize formatting cleanup 2026-04-10 19:49:55 +03:00
Naeel 3b4fe0bb3a security: address medium risks in jwt, redis ordering and body limit 2026-04-10 19:44:34 +03:00
Naeel a5e9bfb15c security: fix critical/high auth, idor, races and persistence 2026-04-10 19:41:59 +03:00
Naeel e9a26f7975 v0.1.17: security hardening — 18/20 vulnerabilities fixed 
Phase 1 (Critical):
- #1 JWT auth (done in v0.1.16)
- #2 Batch message size validation in send_message_batch.go
- #10 RLock in GetQueueUrlV1 (data race fix)

Phase 2 (AWS-compatible limits):
- #3 QueueName validation: max 80 chars, [a-zA-Z0-9_-](.fifo)?
- #4 WaitTimeSeconds clamped to 0-20
- #5 ReceiveMessageWaitTimeSeconds clamped to 0-20
- #6 DelaySeconds clamped to 0-900
- #7 VisibilityTimeout clamped to 0-43200
- #8 MaxNumberOfMessages clamped to 1-10
- #9 Message attributes limited to 10 per message
- #15 BatchEntryId length validated (max 80)
- #16 DeduplicationID length validated (max 128)
- #17 GroupID length validated (max 128)

Phase 3 (Per-tenant resource limits):
- #11 Max messages per queue (120K standard, 20K FIFO)
- #12 Global tenant limit (1000)
- #3.5 HTTP request body size limit (1MB via MaxBytesReader)

Phase 4 (Stability):
- #14 Duplicates map cleanup (already in PeriodicTasks)
- #13 FIFO group lock timeout (already in visibility timeout reset)
- #18 Redis size guard: skip save if >50MB

Skipped (Low, no real risk):
- #19 {account} URL param (informational only, not used for access)
- #20 ReceiptHandle format (self-validating UUID#UUID)

New file: app/gosqs/validation.go — centralized AWS SQS limits and validators
 2026-04-10 18:58:40 +03:00
Naeel b9d434bcc5 v0.1.16: JWT auth via nubes API, auto-provisioning, UI login 
- app/auth/jwt.go: ParseJWTClaims, TenantIDFromSub (sless-compatible SHA256), PingNubesAPI
- app/admin/admin.go: POST /ui/api/auth endpoint, jwtMiddleware for /ui/api/*
- app/tenant/tenant_store.go: NubesSub/Email fields, GetBySub, CreateFromJWT
- app/ui/index.html: login page, email in navbar, JWT session in localStorage
- deployments/k8s/deployment.yaml: v0.1.16, NUBES_ENDPOINT env
- doc/decisions/resource-limits-plan.md: 20 vulnerabilities audit
- Fix: /ui/api/auth moved to subrouter (gorilla/mux PathPrefix conflict)
 2026-04-10 18:35:19 +03:00
Naeel c3ba2dcae4 chore: initial import from sless/shared-sqs (v0.1.14) 
- Standalone SQS-service repository
- Multi-tenant message queue service, AWS SQS compatible
- Based on GoAws, with mutable tenants, auth, WebUI, Redis persistence
- Ready for independent development and deployment
- See doc/ and README.md for architecture and usage
 2026-04-10 16:47:27 +03:00