# Obfuscator

Obfuscator accepts documents in a browser, finds sensitive entities, replaces them consistently across a file set, and returns obfuscated documents plus a separate mapping CSV.

## Runtime Boundary

The application and its upload buffer are separate deployable components:

| Component | Runs on | Responsibility |
| --- | --- | --- |
| Obfuscator Flask application | Managed Flask / Shturval | UI, API, in-memory sessions, document extraction, LLM calls, replacement, and downloads |
| Obfuscator upload buffer | VM213, Nginx WebDAV | Temporary staging of browser uploads until Flask pulls them |

VM213 does not run the Flask application or document-processing code. The buffer is a companion service dedicated to Obfuscator and is not the separate `vmfiles.service` API.

## Request Flow

```text
Browser
	-> GET /                         Flask serves the page
	-> GET /file-picker/...          Flask serves the picker bundle
	-> PUT obfuscator-upload.sinefo.ru/obfuscator-upload/<key>
																			Nginx writes a temporary file on VM213
	-> POST /api/upload_refs         Flask pulls that file with GET, stores bytes
																			in the session, then sends DELETE to VM213
	-> GET /api/process_stream/<sid> Flask runs document processing and streams SSE progress
	-> GET /api/download/<sid>       Browser downloads the obfuscated ZIP
	-> GET /api/csv/<sid>            Browser downloads the mapping separately
```

The browser sends files one at a time: it waits for the WebDAV `PUT`, then asks Flask to pull that file before moving to the next one. The buffer therefore holds files only during transit, not for the complete processing session. If the Flask pull fails, the buffer object may remain until manually removed or otherwise cleaned up; install the TTL cron cleanup described below on VM213.

## Application Components

### Browser UI and picker

- `site/templates/index.html` provides the Obfuscator UI, sets accepted extensions and limits, and configures the VM buffer URL.
- `site/static/dist/file-picker.iife.js` is the browser bundle loaded by the page.
- `upload/frontend/index.js` implements the reusable picker API; `upload/frontend/table/` handles file/folder selection, tree rendering, deduplication, and status updates.
- `upload/frontend/upload/put_to_vm.js` sends a raw file body with `PUT` and reports progress.
- `upload/frontend/upload/upload_via_vm.js` coordinates the per-file `PUT` and `/api/upload_refs` requests.

The UI accepts `.pdf`, `.doc`, `.docx`, `.txt`, `.md`, and `.zip` inputs. CSV input is temporarily disabled in the picker and rejected by both upload endpoints; CSV files inside ZIP archives are not extracted. The CSV parsing and obfuscation code remains in `drhider/extractor.py` for internal use and possible re-enabling after reliability work on the other formats. This does not affect the separate mapping CSV generated as an output for each processing session. ZIP archives are expanded by the browser picker, and their allowed inner files are uploaded individually. The backend `drhider/extractor.py` also expands ZIPs received directly (for example, through `POST /api/upload`). Per-file (50 MiB) and per-session (500 MiB) limits are enforced by the backend; the picker's entry/size/depth limits apply only while unpacking ZIP archives.

### Flask composition and routes

`site/app.py` creates the Flask application and configures request limits and logging. The platform starts the Flask app from `site/`; it does not deploy this VM buffer config.

`site/routes/__init__.py` registers the page, health, processing API, and `upload_refs` blueprints. `site/routes/main_bp.py` serves the page, the picker bundle (`/file-picker/...`), and raw ES modules (`/upload/...`, `/dist/...`). `site/routes/health_bp.py` exposes `/health` with the application version.

Main endpoints:

| Endpoint | Purpose |
| --- | --- |
| `GET /` | Render the browser UI |
| `GET /health` | Liveness and version response |
| `GET /file-picker/<path>` | Serve the built picker bundle |
| `GET /upload/<path>`, `GET /dist/<path>` | Serve raw picker ES modules and dist assets |
| `POST /api/upload_refs` | Pull browser-uploaded buffer objects into a Flask session |
| `POST /api/upload` | Direct multipart upload endpoint |
| `GET /api/session_files/<sid>` | Inspect files in a session (diagnostic endpoint; disabled unless `ENABLE_SESSION_FILES=1`) |
| `GET /api/process_stream/<sid>` | Process a session and stream progress using Server-Sent Events |
| `POST /api/process/<sid>` | Synchronous/legacy processing endpoint |
| `POST /api/cancel/<sid>` | Request a cooperative stop between processing steps |
| `GET /api/download/<sid>` | Download the obfuscated ZIP |
| `GET /api/csv/<sid>` | Download the mapping CSV separately |

### Upload bridge and session state

`upload/backend/upload_refs/blueprint.py` validates names, accepts only URLs under the configured VM-buffer prefix, enforces limits, retries failed pulls, and deletes successfully pulled objects. `pull_file.py` reads the HTTP response in chunks; the resulting file bytes are then kept in the Flask session.

`upload/backend/session/` owns session lifecycle. Sessions use UUID identifiers and process-local RAM; files, the ZIP result, the CSV, and cancellation state are not stored in a database. The configured limits are 50 MiB per file, 500 MiB per session, and a 30-minute TTL. The TTL is paused while processing and resumes afterwards. A Flask process restart loses its sessions and results.

### Document-processing core

`site/routes/api_bp.py` loads all session files and invokes `drhider.obfuscate_files` once for the set. This lets the core use one shared mapping so the same entity receives the same replacement across documents.

The core package is split by responsibility:

- `drhider/extractor.py` converts `.docx`, `.pdf`, and `.doc` content to text/Markdown, expands ZIP archives with ZIP-bomb protections, and retains internal CSV extraction/obfuscation code. CSV input is temporarily disabled at upload entry points while reliability work focuses on other formats; the separate output mapping CSV remains enabled. `.doc` conversion calls the configured conversion service.
- `drhider/scanner.py` detects entities with regular expressions and LLM-based NER, building the original-to-replacement mapping.
- `drhider/replacer.py` applies that mapping to extracted text.
- `drhider/builder.py` builds the obfuscated ZIP and mapping CSV. The mapping CSV is deliberately not included inside the ZIP; it is downloaded through its separate endpoint.
- `drhider/llm_client.py` calls an OpenAI-compatible LLM endpoint and tracks token/time statistics.

Processing runs in a worker thread for the SSE endpoint. Progress, per-file events, LLM activity, completion, cancellation, and errors are sent to the browser as SSE events. Cancellation is cooperative; the current operation/file may finish before the worker stops.

The LLM client reads `LLM_KEY` or `LLM_API_KEY`, with optional `LLM_URL` (default `https://api.aillm.ru/v1/chat/completions`) and `LLM_MODEL` (default `gpt-oss-120b`), from the Managed Flask environment. `LLM_RETRIES` sets the number of retries after the first request (default `3`), and `LLM_RETRY_DELAY` sets the base backoff in seconds (default `2.0`; delays grow linearly). The `.doc` converter URL is configured with `CONVERT_SERVICE_URL` (default `https://liberta.containerk8s.dev.nubes.ru/convert`); `.doc` files are converted sequentially to avoid overloading the conversion service. Secrets belong in service environment settings, never in this repository.

## VM213 WebDAV Buffer

The source of the Obfuscator Nginx config is [`deploy/vm213/nginx/obfuscator-upload.conf`](deploy/vm213/nginx/obfuscator-upload.conf). On VM213 it is installed as `/etc/nginx/sites-enabled/obfuscator-upload.conf`.

- Hostname: `obfuscator-upload.sinefo.ru`
- WebDAV path: `/obfuscator-upload/`
- Disk staging directory: `/var/www/obfuscator-upload/`
- TLS certificate: `/etc/letsencrypt/live/obfuscator-upload.sinefo.ru/`
- Maximum Nginx request body: 1 GiB; the Flask upload bridge is stricter at 50 MiB per file.
- HTTP redirects to HTTPS; the ACME challenge is served from `/var/www/acme`.

The CORS allowlist is for the Obfuscator browser origin `https://obfuscator.pythonk8s.services.ngcloud.ru`. CORS is a browser policy, **not authentication**; direct HTTP clients are not blocked by CORS. The VM disk is temporary staging, not durable storage. A VM or disk failure can lose files that have not yet been pulled by Flask.

Install a cron job on VM213 to remove orphaned upload-buffer files older than 30 minutes. The cleanup command is:

```sh
find /var/www/obfuscator-upload/ -type f -mmin +30 -delete
```

To update the VM config, update this repository source, install the same file on VM213, run `nginx -t`, and reload Nginx only if validation succeeds. The Managed Flask deployment and Nginx deployment are independent; deploying one does not deploy the other.

## Repository Map

```text
site/
	app.py                 Flask app factory and startup entry point
	routes/                UI, health, and processing/upload API blueprints
	templates/             Browser application page
	static/dist/           Built picker bundles served to the browser
upload/
	frontend/              File picker and browser-to-buffer transport
	backend/session/       In-memory sessions, limits, results, and TTL
	backend/upload_refs/   VM-buffer pull bridge
drhider/
	extractor.py           Document and ZIP input handling
	scanner.py             Regex and LLM entity discovery
	replacer.py            Mapping application
	builder.py             ZIP and mapping CSV output
deploy/vm213/nginx/       Source config for the Obfuscator VM213 buffer
```