259 lines
8.5 KiB
Markdown
259 lines
8.5 KiB
Markdown
> ⛔⛔⛔ ЛЕГАСИ (2026-08-16) — СТАРЫЙ IoT (k8s-деплой). НЕ ПРИНИМАТЬ ВО ВНИМАНИЕ. Актуальное: HISTORY/2026-08-16-session-log.md
|
||
|
||
# IoT Managed Service — Актуальная архитектура (v0.2.3)
|
||
|
||
> Дата: 2026-04-12
|
||
> Образ: naeel/iot-operator:v0.2.3
|
||
> Кластер: iot-naeel, namespace: sless
|
||
|
||
---
|
||
|
||
## Путь данных (data flow)
|
||
|
||
```
|
||
IoT Device (MQTT CONNECT)
|
||
| username="{ns}_{deviceId}", password=hex(64)
|
||
v
|
||
EMQX 5.5.1 (Pod emqx, namespace sless)
|
||
| 1. POST /internal/mqtt/auth -> iot-operator:9090
|
||
| - Bridge auth: username=iot-bridge-internal -> allow
|
||
| - Device auth: username={ns}_{deviceId} -> Secret -> compare
|
||
| 2. POST /internal/mqtt/acl -> iot-operator:9090
|
||
| - Device: pub/sub только {ns}/telemetry/{deviceId}
|
||
| - Bridge: sub на +/telemetry/+ (wildcard)
|
||
v
|
||
| MQTT PUBLISH -> topic: "{ns}/telemetry/{deviceId}"
|
||
v
|
||
iot-mqtt-bridge (Pod, namespace sless)
|
||
| paho.mqtt.golang, подписка на "+/telemetry/+"
|
||
| Парсит topic -> namespace (segment 0), deviceId (segment 2)
|
||
| AWS SDK SQS SendMessage -> shared-SQS
|
||
v
|
||
shared-SQS (namespace shared-sqs)
|
||
| Endpoint: https://qu.kube5s.ru
|
||
| Tenant: iot-service (ID: t-96afe7e9f781f6ca)
|
||
| Queue: iot-telemetry
|
||
v
|
||
iot-sqs-consumer (Pod, namespace sless)
|
||
| ReceiveMessage (WaitTimeSeconds=20, long polling)
|
||
| Парсит envelope -> namespace, device_id, payload
|
||
| EnsureTenantDB(namespace) -> CREATE DATABASE tenant_{ns}
|
||
| INSERT INTO iot_telemetry
|
||
| DeleteMessage (at-least-once)
|
||
v
|
||
Managed PostgreSQL 17
|
||
| Host: postgresqlk8s-master.dc5db45d-....svc.cluster.local
|
||
| Per-tenant: DATABASE tenant_{namespace}
|
||
| Таблица: iot_telemetry (id, device_id, payload JSONB, received_at, created_at)
|
||
v
|
||
REST API (iot-operator:9090)
|
||
| GET /v1/namespaces/{ns}/iot/telemetry?device=X&limit=N
|
||
v
|
||
Пользователь
|
||
| IoT Console: https://iot.kube5s.ru/console
|
||
| WebSocket MQTT: wss://iot.kube5s.ru/mqtt
|
||
| Terraform: sless_iot_device resource
|
||
```
|
||
|
||
---
|
||
|
||
## Компоненты
|
||
|
||
### iot-operator (cmd/iot-operator)
|
||
|
||
Единый бинарник: controller-manager + REST API сервер.
|
||
|
||
| Функция | Описание |
|
||
|---------|----------|
|
||
| IoTDevice Controller | Reconcile: создание Secret с MQTT credentials, OwnerReference |
|
||
| REST API :9090 | CRUD устройств, телеметрия, MQTT auth/acl, admin stats, UI |
|
||
| Health :8081 | /healthz, /readyz для k8s probes |
|
||
|
||
### iot-mqtt-bridge (cmd/mqtt-bridge)
|
||
|
||
MQTT subscriber -> SQS producer. Stateless.
|
||
|
||
| Параметр | Значение |
|
||
|----------|----------|
|
||
| MQTT broker | tcp://emqx.sless.svc:1883 |
|
||
| MQTT username | iot-bridge-internal (Secret iot-bridge-credentials) |
|
||
| MQTT subscription | +/telemetry/+ |
|
||
| SQS endpoint | https://qu.kube5s.ru |
|
||
| SQS queue | iot-telemetry |
|
||
|
||
Bridge auth (v0.2.3): operator проверяет BridgeUsername/BridgePassword ДО парсинга namespace_deviceId.
|
||
|
||
### iot-sqs-consumer (cmd/sqs-consumer)
|
||
|
||
SQS consumer -> Postgres writer. Stateless.
|
||
|
||
| Параметр | Значение |
|
||
|----------|----------|
|
||
| SQS endpoint | https://qu.kube5s.ru |
|
||
| SQS queue | iot-telemetry |
|
||
| Long polling | WaitTimeSeconds=20 |
|
||
| Postgres | IOT_PG_DSN из Secret |
|
||
| Семантика | at-least-once (DeleteMessage после INSERT) |
|
||
|
||
### EMQX 5.5.1
|
||
|
||
MQTT-брокер с HTTP auth backend.
|
||
|
||
| Параметр | Значение |
|
||
|----------|----------|
|
||
| Образ | emqx/emqx:5.5.1 |
|
||
| Порты | 1883 (MQTT), 8083 (WebSocket), 18083 (Dashboard) |
|
||
| Auth | HTTP POST -> iot-operator:9090/internal/mqtt/auth |
|
||
| ACL | HTTP POST -> iot-operator:9090/internal/mqtt/acl |
|
||
| Конфиг | HOCON emqx.conf через ConfigMap |
|
||
|
||
---
|
||
|
||
## CRD: IoTDevice (iot.kube5s.ru/v1alpha1)
|
||
|
||
### Spec
|
||
|
||
| Поле | Тип | Обязательное | Описание |
|
||
|------|-----|-------------|----------|
|
||
| deviceId | string | да | Pattern: ^[a-z0-9][a-z0-9-]*[a-z0-9]$ |
|
||
| enabled | bool | нет | default: true |
|
||
| metadata | map[string]string | нет | Произвольные метаданные |
|
||
|
||
### Status
|
||
|
||
| Поле | Описание |
|
||
|------|----------|
|
||
| phase | Active / Disabled / Pending / Error |
|
||
| mqttUsername | {namespace}_{deviceId} |
|
||
| secretName | iot-{deviceId} |
|
||
| topicPrefix | {namespace}/ |
|
||
| message | Сообщение об ошибке |
|
||
|
||
### Reconcile logic
|
||
|
||
1. Добавить finalizer iot.kube5s.ru/device-cleanup
|
||
2. Если Secret iot-{deviceId} не существует:
|
||
- crypto/rand 32 bytes -> hex (64 символа) = пароль
|
||
- Создать Secret с OwnerReference -> каскадное удаление
|
||
- Keys: mqtt-username, mqtt-password, device-id
|
||
3. Status: phase=Active, mqttUsername={ns}_{deviceId}
|
||
4. Если enabled=false -> phase=Disabled (Secret НЕ удаляется)
|
||
5. DELETE: finalizer cleanup -> Secret удаляется каскадно
|
||
|
||
---
|
||
|
||
## Аутентификация
|
||
|
||
### REST API (/v1/)
|
||
|
||
- Middleware: Bearer JWT token
|
||
- authTestMode = true (текущий): любой непустой Bearer token принимается
|
||
- authTestMode = false (prod): JWT decode -> sub -> SHA256 -> namespace mapping
|
||
|
||
### MQTT Auth (/internal/mqtt/auth)
|
||
|
||
1. Если username == BridgeUsername -> проверить BridgePassword -> allow/deny
|
||
2. Иначе: парсить username по первому "_" -> namespace + deviceId
|
||
3. Найти Secret iot-{deviceId} в namespace
|
||
4. crypto/subtle.ConstantTimeCompare(password, secret.mqtt-password)
|
||
5. Всегда HTTP 200, body: {"result": "allow"} или {"result": "deny"}
|
||
|
||
### MQTT ACL (/internal/mqtt/acl)
|
||
|
||
- Bridge (clientid=sless-iot-bridge): allow subscribe +/telemetry/+
|
||
- Device: allow pub/sub только {namespace}/telemetry/{deviceId}
|
||
- Всё остальное: deny
|
||
|
||
---
|
||
|
||
## Хранение данных
|
||
|
||
### PostgreSQL (managed)
|
||
|
||
| Параметр | Значение |
|
||
|----------|----------|
|
||
| Версия | PG 17 |
|
||
| Host | postgresqlk8s-master.dc5db45d-f8b4-4fd0-ad33-ec4dd017f2d5.svc.cluster.local |
|
||
| User | super |
|
||
| Master DB | sqsdb |
|
||
|
||
Per-tenant изоляция: отдельная DATABASE tenant_{namespace}.
|
||
|
||
Таблица iot_telemetry:
|
||
- id SERIAL PRIMARY KEY
|
||
- device_id TEXT NOT NULL
|
||
- payload JSONB NOT NULL
|
||
- received_at TIMESTAMPTZ
|
||
- created_at TIMESTAMPTZ DEFAULT NOW()
|
||
|
||
### shared-SQS
|
||
|
||
| Параметр | Значение |
|
||
|----------|----------|
|
||
| Endpoint | https://qu.kube5s.ru |
|
||
| Tenant | iot-service (t-96afe7e9f781f6ca) |
|
||
| Queue | iot-telemetry |
|
||
| Протокол | AWS SQS API compatible |
|
||
|
||
---
|
||
|
||
## Docker образ
|
||
|
||
- Registry: Docker Hub naeel/iot-operator
|
||
- Базовый: gcr.io/distroless/static:nonroot
|
||
- Содержит 3 бинарника: /iot-operator, /mqtt-bridge, /sqs-consumer
|
||
- Выбор бинарника через command в Deployment YAML
|
||
|
||
---
|
||
|
||
## Сетевая схема
|
||
|
||
```
|
||
Internet
|
||
|
|
||
v
|
||
nginx-ingress (namespace ingress)
|
||
| iot.kube5s.ru/mqtt -> emqx-ws:8083 (WebSocket)
|
||
| iot.kube5s.ru/console -> iot-operator:9090 (UI)
|
||
| iot.kube5s.ru/iot-admin -> iot-operator:9090 (Admin UI)
|
||
v
|
||
namespace sless:
|
||
emqx:1883 <-> iot-mqtt-bridge (MQTT)
|
||
emqx:1883 <- IoT devices (MQTT)
|
||
iot-operator:9090 <- emqx (auth/acl HTTP)
|
||
iot-operator:9090 <- users (REST API)
|
||
iot-mqtt-bridge -> shared-sqs (HTTPS, SQS API)
|
||
iot-sqs-consumer <- shared-sqs (HTTPS, SQS API)
|
||
iot-sqs-consumer -> managed-postgres (TCP 5432)
|
||
iot-operator -> managed-postgres (TCP 5432, telemetry GET)
|
||
iot-operator -> k8s API (CRD watch, Secret CRUD)
|
||
|
||
namespace shared-sqs:
|
||
shared-sqs:9324 (SQS API)
|
||
Ingress: qu.kube5s.ru -> shared-sqs
|
||
|
||
namespace dc5db45d-...:
|
||
postgresqlk8s-0 (PG 17 managed)
|
||
```
|
||
|
||
---
|
||
|
||
## Секреты (namespace sless)
|
||
|
||
| Secret | Ключи | Используется |
|
||
|--------|-------|-------------|
|
||
| iot-bridge-credentials | MQTT_USERNAME, MQTT_PASSWORD | iot-operator, iot-mqtt-bridge |
|
||
| iot-sqs-credentials | SQS_ENDPOINT, SQS_ACCESS_KEY, SQS_SECRET_KEY | iot-mqtt-bridge, iot-sqs-consumer |
|
||
| iot-postgres-secret | IOT_PG_DSN | iot-sqs-consumer |
|
||
|
||
---
|
||
|
||
## Эволюция архитектуры
|
||
|
||
| Версия | Дата | Message bus | Postgres | Кластер |
|
||
|--------|------|------------|----------|---------|
|
||
| v0.1.50 | 2026-04-04 | RabbitMQ (AMQP) | emptyDir PVC | sless (общий) |
|
||
| v0.1.68 | 2026-04-06 | Kafka (segmentio/kafka-go) | emptyDir PVC | sless (общий) |
|
||
| v0.2.0 | 2026-04-12 | shared-SQS (AWS SDK) | Managed PG 17 | iot-naeel (новый) |
|
||
| v0.2.3 | 2026-04-12 | shared-SQS (AWS SDK) | Managed PG 17 | iot-naeel |
|