Start here: devops/README.md (this file). 1a) Architecture: devops/ARCHITECTURE.md. 1b) Mandatory suspend lifecycle policy: docs/60_strategy/provider_philosophy.md (sections 7-9).
Service list: devops/config/services_list.txt. 2a) Operation timeouts config: devops/config/operation_timeouts.json.
Unified YAML generator: universal_rebuild/tools/service_spec_gen.
Go resource generator: universal_rebuild/tools/gen_v2.
Docs generator: devops/02_generate_resources_and_docs_v2.sh.
Build + publish scripts: devops/03_build_and_upload_provider.sh, devops/04_build_and_publish_docs.sh.

Overview (end-to-end)

Fetch all service metadata from API
Generate unified YAML specs from API
Generate Go resources + documentation files from unified YAML
Build provider binaries for 3 OS targets and upload to Registry
Build and publish documentation site

Generator code locations

Unified YAML from API: universal_rebuild/tools/service_spec_gen
Go resources from YAML: universal_rebuild/tools/gen_v2
Docs generator: devops/02_generate_resources_and_docs_v2.sh

Prerequisites

Go 1.22+
python3
gpg
mc (MinIO/S3 client)
Docker (for mkdocs build)

Shared settings

S3 environment:

S3_ENDPOINT (example: https://s3.msk-1.ngcloud.ru)
S3_ACCESS_KEY
S3_SECRET_KEY

Tip: the local S3 config lives at secrets/.s3cfg_registry.

Operation timeouts config:

Source of truth: devops/config/operation_timeouts.json
During provider build, this file is copied into embedded provider config by devops/03_build_and_upload_provider.sh.

Provider naming defaults:

REGISTRY_HOSTNAME: terra.k8c.ru
NAMESPACE: nubes
NAME: nubes

Stand profiles (TEST/PROD/DEV)

To avoid overwrite between stands, use per-stand profiles:

devops/profiles/test
devops/profiles/prod
devops/profiles/dev

Each profile contains:

profile.env (API URL, token file, versions, S3/GPG paths)
services_list.txt
operation_timeouts.json
generated/ (resources_yaml, go, docs, json, tmp)

Run scripts with profile path:

./devops/01_generate_yamls.sh --profile devops/profiles/test
./devops/02_generate_resources_and_docs_v2.sh --profile devops/profiles/test
./devops/03_build_and_upload_provider.sh --profile devops/profiles/test
./devops/04_build_and_publish_docs.sh --profile devops/profiles/test

--profile is required for generation/build scripts in strict mode. Generated artifacts are stored only in devops/profiles/<stand>/generated/*. universal_rebuild is treated as code-only source directory.

Step 1: Generate unified YAML from API

Script: 01_generate_yamls.sh

Input list of services:

config/services_list.txt (service_id only)

Token options:

TOKEN_FILE=<repo_root>/HH-MM-SS.token, or
NUBES_API_TOKEN directly

Example:

export TOKEN_FILE=<repo_root>/08-33-41.token
./devops/01_generate_yamls.sh

Outputs:

YAML files in universal_rebuild/resources_yaml (file names use ID_name.yaml)

Note:

Every run regenerates all YAML specs from the API for the services listed in services_list.txt.
This overwrites previous YAMLs and ensures resources/docs match the current API for that list.
Before generation, the scripts clean old YAML and generated Go files, so each run starts from a clean slate.

Step 2: Generate Go resources and docs

Script: 02_generate_resources_and_docs_v2.sh

Example:

./devops/02_generate_resources_and_docs_v2.sh

Outputs:

Go files in universal_rebuild/internal/resources_gen
Docs in docs/30_registry/resources

Step 3: Build and upload provider

Script: 03_build_and_upload_provider.sh

Uses registry-server-build/build-provider.sh and signs with:

secrets/private_key.asc (ignored by git)

Example:

export S3_ENDPOINT=https://s3.msk-1.ngcloud.ru
export S3_ACCESS_KEY=...
export S3_SECRET_KEY=...
./devops/03_build_and_upload_provider.sh 2.0.2

Step 4: Build and publish docs

Script: 04_build_and_publish_docs.sh

Example:

export S3_ENDPOINT=https://s3.msk-1.ngcloud.ru
export S3_ACCESS_KEY=...
export S3_SECRET_KEY=...
./devops/04_build_and_publish_docs.sh 2.0.2

Notes

The GPG private key must remain stable across releases. Do not regenerate per build.
If the key is regenerated, the registry server must be updated to serve the new public key.
Terraform will fail with authentication signature from unknown issuer if the registry public key does not match the signing key.
config/services_list.txt is the source of truth for which services are generated.
If the provider version changes, update universal_rebuild/main.go.

Quickstart (fresh run)

# 0) Prepare token
export TOKEN_FILE=<repo_root>/HH-MM-SS.token

# 1) Generate YAML specs from API
./devops/01_generate_yamls.sh

# 2) Generate Go resources + docs
./devops/02_generate_resources_and_docs_v2.sh

# 3) Build + upload provider
./devops/03_build_and_upload_provider.sh 2.0.2

# 4) Build + publish docs
./devops/04_build_and_publish_docs.sh 2.0.2

Troubleshooting checklist

If YAML generation fails: check token, API endpoint, and /tmp/yaml_gen_failures.txt.
If docs are missing: run 02_generate_resources_and_docs_v2.sh.
If registry rejects artifacts: verify GPG key and public key in registry server.

One-time GPG bootstrap (do this once, keep the key stable)

Generate and export keys (no passphrase):

GPG_DIR=<repo_root>/secrets
GNUPGHOME=$(mktemp -d)
cat > /tmp/gpg_batch <<'EOF'
%no-protection
Key-Type: RSA
Key-Length: 4096
Subkey-Type: RSA
Subkey-Length: 4096
Name-Real: tazet@narod.ru
Name-Email: tazet@narod.ru
Expire-Date: 0
EOF
gpg --batch --homedir "$GNUPGHOME" --gen-key /tmp/gpg_batch
gpg --batch --homedir "$GNUPGHOME" --armor --export-secret-keys > "$GPG_DIR/private_key.asc"
gpg --batch --homedir "$GNUPGHOME" --armor --export > "$GPG_DIR/public_key.asc"
rm -rf "$GNUPGHOME" /tmp/gpg_batch

Update registry server public key (ASCII Armor) in:

registry-server-build/main.go
operator/cmd/registry/main.go

Rebuild and redeploy the registry server (see docs/50_history/00_system_mechanics.md).
Build and upload provider artifacts as usual.

Token and self-run shortcuts

Token capture (JSON from F12):

cat token.json | ./devops/00_token_manager.sh save -

Use latest token automatically:

./devops/12_generate_yamls_latest.sh

10-run stability test (latest token):

REQUEST_DELAY=0.3 ATTEMPTS=3 ./devops/11_yaml_stability_run_latest.sh

Log output:

tail -n 200 /tmp/yaml_gen_runs.log

Legacy (deprecated)

Ops YAML and ops-doc generators were removed after the unified YAML migration.

Docs build policy (updated)

mkdocs/mkdocs-material are not installed on every docs generation run.
Preferred path: build docs in Docker (squidfunk/mkdocs-material) via 04_build_and_publish_docs.sh.
Fallback path: local mkdocs is used only if it is already installed.
If local mkdocs is missing, install once in persistent virtualenv and reuse.

One-time setup example:

python3 -m venv .venv
.venv/bin/pip install mkdocs mkdocs-material

Then run docs publish normally:

./devops/04_build_and_publish_docs.sh <version>