diff --git a/TOOLS/README.md b/TOOLS/README.md new file mode 100644 index 0000000..e606e7f --- /dev/null +++ b/TOOLS/README.md @@ -0,0 +1,41 @@ +# TOOLS — Генераторы Terraform-провайдера Nubes + +Каждый инструмент — независимый Go-модуль. + +## yaml-generator + +API Nubes → `resources_yaml/*.yaml` + +```bash +cd yaml-generator && go build -o ../bin/yaml-generator . +./bin/yaml-generator +``` + +Структура: `main.go` + `internal/{client,config,normalize,spec,types}`. + +## resource-generator + +`resources_yaml/*.yaml` → `internal/resources_gen/*.go` + `registry.go` + +```bash +cd resource-generator && go build -o ../bin/resource-generator . +./bin/resource-generator +``` + +Структура: `main.go` + `internal/{helpers,loader,params,templates,types,writers}`. + +## docs-generator + +`resources_yaml/*.yaml` → Markdown-документация в `docs/30_registry/resources/` + +```bash +cd docs-generator && go build -o ../bin/docs-generator . + +# Документация ресурсов +./bin/docs-generator + +# Operations-документация +./bin/docs-generator --ops +``` + +Структура: `main.go` + `internal/{types,writers,ops}`. diff --git a/devops/README.md b/devops/README.md index 824da4e..757e8ee 100644 --- a/devops/README.md +++ b/devops/README.md @@ -1,103 +1,91 @@ -# DevOps Runbook: Provider Build Pipeline +# DevOps — Pipeline сборки Terraform-провайдера Nubes -This folder describes the full pipeline from API discovery to provider + docs publication. +## Обзор -## Read This First +``` +API Nubes → YAML → Go-ресурсы → Провайдер → S3 Registry + ↘ Markdown-доки → MkDocs-сайт +``` -1) Start here: `devops/README.md` (this file). -1a) Architecture: `devops/ARCHITECTURE.md`. -1b) Mandatory suspend lifecycle policy: `docs/60_strategy/provider_philosophy.md` (sections 7-9). -2) Service list: `devops/config/services_list.txt`. -2a) Operation timeouts config: `devops/config/operation_timeouts.json`. -3) Unified YAML generator: `universal_rebuild/tools/service_spec_gen`. -4) Go resource generator: `universal_rebuild/tools/gen_v2`. -5) Docs generator: `devops/02_generate_resources_and_docs_v2.sh`. -6) Build + publish scripts: `devops/03_build_and_upload_provider.sh`, `devops/04_build_and_publish_docs.sh`. +## Инструменты (TOOLS/) -## Overview (end-to-end) +| Бинарник | Исходники | Назначение | +|---|---|---| +| `TOOLS/bin/yaml-generator` | `TOOLS/yaml-generator/` | API → `resources_yaml/*.yaml` | +| `TOOLS/bin/resource-generator` | `TOOLS/resource-generator/` | YAML → `internal/resources_gen/*.go` | +| `TOOLS/bin/docs-generator` | `TOOLS/docs-generator/` | YAML → Markdown-документация | -1) Fetch all service metadata from API -2) Generate unified YAML specs from API -3) Generate Go resources + documentation files from unified YAML -4) Build provider binaries for 3 OS targets and upload to Registry -5) Build and publish documentation site +Каждый инструмент — независимый Go-модуль со своим `go.mod`. -## Generator code locations +## Скрипты пайплайна -- Unified YAML from API: `universal_rebuild/tools/service_spec_gen` -- Go resources from YAML: `universal_rebuild/tools/gen_v2` -- Docs generator: `devops/02_generate_resources_and_docs_v2.sh` +| # | Скрипт | Что делает | +|---|---|---| +| 00 | `00_token_manager.sh` | Получение/обновление API-токенов | +| 01 | `01_generate_yamls.sh` | API → YAML (требует `--profile`) | +| 02 | `02_generate_resources_and_docs_v2.sh` | YAML → Go + Docs (требует `--profile`) | +| 03 | `03_build_and_upload_provider.sh` | Сборка провайдера → GPG → S3 | +| 04 | `04_build_and_publish_docs.sh` | MkDocs-сайт → публикация | +| — | `10_yaml_stability_run.sh` | Тест стабильности (N запусков 01) | +| — | `11_yaml_stability_run_latest.sh` | Авто-токен → 10 | +| — | `12_generate_yamls_latest.sh` | Авто-токен → 01 | +| — | `13_generate_yamls_clean.sh` | Очистка YAML + авто-токен → 01 | +| — | `build-provider.sh` | Быстрая локальная сборка (без S3) | -## Prerequisites +## Профили стендов -- Go 1.22+ -- `python3` -- `gpg` -- `mc` (MinIO/S3 client) -- Docker (for mkdocs build) +``` +devops/profiles/ +├── test/profile.env # lk-api-gateway-test.ngcloud.ru +├── prod/profile.env # продакшн +└── dev/profile.env # дев-стенд +``` -## Shared settings +Каждый профиль: свой endpoint, токен, версия, S3-бакет, GPG-ключи. -S3 environment: -- `S3_ENDPOINT` (example: `https://s3.msk-1.ngcloud.ru`) -- `S3_ACCESS_KEY` -- `S3_SECRET_KEY` - -Tip: the local S3 config lives at `secrets/.s3cfg_registry`. - -Operation timeouts config: -- Source of truth: `devops/config/operation_timeouts.json` -- During provider build, this file is copied into embedded provider config by `devops/03_build_and_upload_provider.sh`. - -Provider naming defaults: -- `REGISTRY_HOSTNAME`: `terra.k8c.ru` -- `NAMESPACE`: `nubes` -- `NAME`: `nubes` - -## Stand profiles (TEST/PROD/DEV) - -To avoid overwrite between stands, use per-stand profiles: - -- `devops/profiles/test` -- `devops/profiles/prod` -- `devops/profiles/dev` - -Each profile contains: - -- `profile.env` (API URL, token file, versions, S3/GPG paths) -- `services_list.txt` -- `operation_timeouts.json` -- `generated/` (`resources_yaml`, `go`, `docs`, `json`, `tmp`) - -Run scripts with profile path: +## Полный пайплайн (пример для test-стенда) ```bash +# 1. Токен +./devops/00_token_manager.sh --profile devops/profiles/test + +# 2. YAML из API ./devops/01_generate_yamls.sh --profile devops/profiles/test + +# 3. Go-ресурсы + документация ./devops/02_generate_resources_and_docs_v2.sh --profile devops/profiles/test + +# 4. Сборка и загрузка провайдера ./devops/03_build_and_upload_provider.sh --profile devops/profiles/test + +# 5. Публикация документации ./devops/04_build_and_publish_docs.sh --profile devops/profiles/test ``` -`--profile` is required for generation/build scripts in strict mode. -Generated artifacts are stored only in `devops/profiles//generated/*`. -`universal_rebuild` is treated as code-only source directory. +## Структура проекта -## Step 1: Generate unified YAML from API - -Script: `01_generate_yamls.sh` - -Input list of services: -- `config/services_list.txt` (service_id only) - -Token options: -- `TOKEN_FILE=/HH-MM-SS.token`, or -- `NUBES_API_TOKEN` directly - -Example: -```bash -export TOKEN_FILE=/08-33-41.token -./devops/01_generate_yamls.sh ``` +tf_provider/ +├── TOOLS/ # Генераторы (независимые Go-модули) +│ ├── yaml-generator/ +│ ├── resource-generator/ +│ ├── docs-generator/ +│ └── bin/ # Скомпилированные бинарники +├── provider/ # Terraform-провайдер (Go-модуль) +│ ├── internal/ +│ │ ├── core/ # HTTP-клиент API +│ │ ├── provider/ # Регистрация провайдера +│ │ ├── resources_core/ # Общая логика CRUD +│ │ └── resources_gen/ # Сгенерированные ресурсы +│ └── main.go +└── devops/ # Скрипты пайплайна + профили + ├── profiles/ + └── config/ +``` + +## Архитектура + +Подробно: `devops/ARCHITECTURE.md`. Outputs: - YAML files in `universal_rebuild/resources_yaml` (file names use `ID_name.yaml`)