tf_provider/devops/README.md

# DevOps — Pipeline сборки Terraform-провайдера Nubes

## Обзор

```
API Nubes → YAML → Go-ресурсы → Провайдер → S3 Registry
                    ↘ Markdown-доки → MkDocs-сайт
```

## Инструменты (TOOLS/)

| Бинарник | Исходники | Назначение |
|---|---|---|
| `TOOLS/bin/yaml-generator` | `TOOLS/yaml-generator/` | API → `resources_yaml/*.yaml` |
| `TOOLS/bin/resource-generator` | `TOOLS/resource-generator/` | YAML → `internal/resources_gen/*.go` |
| `TOOLS/bin/docs-generator` | `TOOLS/docs-generator/` | YAML → Markdown-документация |

Каждый инструмент — независимый Go-модуль со своим `go.mod`.

## Скрипты пайплайна

| # | Скрипт | Что делает |
|---|---|---|
| 00 | `00_token_manager.sh` | Получение/обновление API-токенов |
| 01 | `01_generate_yamls.sh` | API → YAML (требует `--profile`) |
| 02 | `02_generate_resources_and_docs_v2.sh` | YAML → Go + Docs (требует `--profile`) |
| 03 | `03_build_and_upload_provider.sh` | Сборка провайдера → GPG → S3 |
| 04 | `04_build_and_publish_docs.sh` | MkDocs-сайт → публикация |
| — | `10_yaml_stability_run.sh` | Тест стабильности (N запусков 01) |
| — | `11_yaml_stability_run_latest.sh` | Авто-токен → 10 |
| — | `12_generate_yamls_latest.sh` | Авто-токен → 01 |
| — | `13_generate_yamls_clean.sh` | Очистка YAML + авто-токен → 01 |
| — | `build-provider.sh` | Быстрая локальная сборка (без S3) |

## Профили стендов

```
devops/profiles/
├── test/profile.env    # lk-api-gateway-test.ngcloud.ru
├── prod/profile.env    # продакшн
└── dev/profile.env     # дев-стенд
```

Каждый профиль: свой endpoint, токен, версия, S3-бакет, GPG-ключи.

## Полный пайплайн (пример для test-стенда)

```bash
# 1. Токен
./devops/00_token_manager.sh --profile devops/profiles/test

# 2. YAML из API
./devops/01_generate_yamls.sh --profile devops/profiles/test

# 3. Go-ресурсы + документация
./devops/02_generate_resources_and_docs_v2.sh --profile devops/profiles/test

# 4. Сборка и загрузка провайдера
./devops/03_build_and_upload_provider.sh --profile devops/profiles/test

# 5. Публикация документации
./devops/04_build_and_publish_docs.sh --profile devops/profiles/test
```

## Структура проекта

```
tf_provider/
├── TOOLS/               # Генераторы (независимые Go-модули)
│   ├── yaml-generator/
│   ├── resource-generator/
│   ├── docs-generator/
│   └── bin/              # Скомпилированные бинарники
├── provider/             # Terraform-провайдер (Go-модуль)
│   ├── internal/
│   │   ├── core/         # HTTP-клиент API
│   │   ├── provider/     # Регистрация провайдера
│   │   ├── resources_core/ # Общая логика CRUD
│   │   └── resources_gen/  # Сгенерированные ресурсы
│   └── main.go
└── devops/               # Скрипты пайплайна + профили
    ├── profiles/
    └── config/
```

## Архитектура

Подробно: `devops/ARCHITECTURE.md`.

Outputs:
- YAML files in `universal_rebuild/resources_yaml` (file names use `ID_name.yaml`)

Note:
- Every run regenerates all YAML specs from the API for the services listed in `services_list.txt`.
- This overwrites previous YAMLs and ensures resources/docs match the current API for that list.
- Before generation, the scripts clean old YAML and generated Go files, so each run starts from a clean slate.

## Step 2: Generate Go resources and docs

Script: `02_generate_resources_and_docs_v2.sh`

Example:
```bash
./devops/02_generate_resources_and_docs_v2.sh
```

Outputs:
- Go files in `universal_rebuild/internal/resources_gen`
- Docs in `docs/30_registry/resources`

## Step 3: Build and upload provider

Script: `03_build_and_upload_provider.sh`

Uses `registry-server-build/build-provider.sh` and signs with:
- `secrets/private_key.asc` (ignored by git)

Example:
```bash
export S3_ENDPOINT=https://s3.msk-1.ngcloud.ru
export S3_ACCESS_KEY=...
export S3_SECRET_KEY=...
./devops/03_build_and_upload_provider.sh 2.0.2
```

## Step 4: Build and publish docs

Script: `04_build_and_publish_docs.sh`

Example:
```bash
export S3_ENDPOINT=https://s3.msk-1.ngcloud.ru
export S3_ACCESS_KEY=...
export S3_SECRET_KEY=...
./devops/04_build_and_publish_docs.sh 2.0.2
```

## Notes

- The GPG private key must remain stable across releases. Do not regenerate per build.
- If the key is regenerated, the registry server must be updated to serve the new public key.
- Terraform will fail with `authentication signature from unknown issuer` if the registry public key does not match the signing key.
- `config/services_list.txt` is the source of truth for which services are generated.
- If the provider version changes, update `universal_rebuild/main.go`.

## Quickstart (fresh run)

```bash
# 0) Prepare token
export TOKEN_FILE=<repo_root>/HH-MM-SS.token

# 1) Generate YAML specs from API
./devops/01_generate_yamls.sh

# 2) Generate Go resources + docs
./devops/02_generate_resources_and_docs_v2.sh

# 3) Build + upload provider
./devops/03_build_and_upload_provider.sh 2.0.2

# 4) Build + publish docs
./devops/04_build_and_publish_docs.sh 2.0.2
```

## Troubleshooting checklist

- If YAML generation fails: check token, API endpoint, and `/tmp/yaml_gen_failures.txt`.
- If docs are missing: run `02_generate_resources_and_docs_v2.sh`.
- If registry rejects artifacts: verify GPG key and public key in registry server.

## One-time GPG bootstrap (do this once, keep the key stable)

1) Generate and export keys (no passphrase):
```bash
GPG_DIR=<repo_root>/secrets
GNUPGHOME=$(mktemp -d)
cat > /tmp/gpg_batch <<'EOF'
%no-protection
Key-Type: RSA
Key-Length: 4096
Subkey-Type: RSA
Subkey-Length: 4096
Name-Real: tazet@narod.ru
Name-Email: tazet@narod.ru
Expire-Date: 0
EOF
gpg --batch --homedir "$GNUPGHOME" --gen-key /tmp/gpg_batch
gpg --batch --homedir "$GNUPGHOME" --armor --export-secret-keys > "$GPG_DIR/private_key.asc"
gpg --batch --homedir "$GNUPGHOME" --armor --export > "$GPG_DIR/public_key.asc"
rm -rf "$GNUPGHOME" /tmp/gpg_batch
```

2) Update registry server public key (ASCII Armor) in:
- `registry-server-build/main.go`
- `operator/cmd/registry/main.go`

3) Rebuild and redeploy the registry server (see `docs/50_history/00_system_mechanics.md`).

4) Build and upload provider artifacts as usual.

## Token and self-run shortcuts

Token capture (JSON from F12):
```bash
cat token.json | ./devops/00_token_manager.sh save -
```

Use latest token automatically:
```bash
./devops/12_generate_yamls_latest.sh
```

10-run stability test (latest token):
```bash
REQUEST_DELAY=0.3 ATTEMPTS=3 ./devops/11_yaml_stability_run_latest.sh
```

Log output:
```bash
tail -n 200 /tmp/yaml_gen_runs.log
```

## Legacy (deprecated)

- Ops YAML and ops-doc generators were removed after the unified YAML migration.

## Docs build policy (updated)

- `mkdocs`/`mkdocs-material` are **not** installed on every docs generation run.
- Preferred path: build docs in Docker (`squidfunk/mkdocs-material`) via `04_build_and_publish_docs.sh`.
- Fallback path: local `mkdocs` is used only if it is already installed.
- If local `mkdocs` is missing, install once in persistent virtualenv and reuse.

One-time setup example:

```bash
python3 -m venv .venv
.venv/bin/pip install mkdocs mkdocs-material
```

Then run docs publish normally:

```bash
./devops/04_build_and_publish_docs.sh <version>
```