obfuscator/README.md

Obfuscator

Obfuscator accepts documents in a browser, finds sensitive entities, replaces them consistently across a file set, and returns obfuscated documents plus a separate mapping CSV.

Runtime Boundary

The application and its upload buffer are separate deployable components:

Component	Runs on	Responsibility
Obfuscator Flask application	Managed Flask / Shturval	UI, API, in-memory sessions, document extraction, LLM calls, replacement, and downloads
Obfuscator upload buffer	VM213, Nginx WebDAV	Temporary staging of browser uploads until Flask pulls them

VM213 does not run the Flask application or document-processing code. The buffer is a companion service dedicated to Obfuscator and is not the separate vmfiles.service API.

Request Flow

Browser
	-> GET /                         Flask serves the page
	-> GET /file-picker/...          Flask serves the picker bundle
	-> PUT obfuscator-upload.sinefo.ru/obfuscator-upload/<key>
																			Nginx writes a temporary file on VM213
	-> POST /api/upload_refs         Flask pulls that file with GET, stores bytes
																			in the session, then sends DELETE to VM213
	-> GET /api/process_stream/<sid> Flask runs document processing and streams SSE progress
	-> GET /api/download/<sid>       Browser downloads the obfuscated ZIP
	-> GET /api/csv/<sid>            Browser downloads the mapping separately

The browser sends files one at a time: it waits for the WebDAV PUT, then asks Flask to pull that file before moving to the next one. The buffer therefore holds files only during transit, not for the complete processing session. If the Flask pull fails, the buffer object may remain until manually removed or otherwise cleaned up; install the TTL cron cleanup described below on VM213.

Application Components

Browser UI and picker

• site/templates/index.html provides the Obfuscator UI, sets accepted extensions and limits, and configures the VM buffer URL.
• site/static/dist/file-picker.iife.js is the browser bundle loaded by the page.
• upload/frontend/index.js implements the reusable picker API; upload/frontend/table/ handles file/folder selection, tree rendering, deduplication, and status updates.
• upload/frontend/upload/put_to_vm.js sends a raw file body with PUT and reports progress.
• upload/frontend/upload/upload_via_vm.js coordinates the per-file PUT and /api/upload_refs requests.

The UI accepts .pdf, .doc, .docx, .txt, .md, and .zip inputs. CSV input is temporarily disabled in the picker and rejected by both upload endpoints; CSV files inside ZIP archives are not extracted. The CSV parsing and obfuscation code remains in drhider/extractor.py for internal use and possible re-enabling after reliability work on the other formats. This does not affect the separate mapping CSV generated as an output for each processing session. ZIP archives are expanded by the browser picker, and their allowed inner files are uploaded individually. The backend drhider/extractor.py also expands ZIPs received directly (for example, through POST /api/upload). Per-file (50 MiB) and per-session (500 MiB) limits are enforced by the backend; the picker's entry/size/depth limits apply only while unpacking ZIP archives.

Flask composition and routes

site/app.py creates the Flask application and configures request limits and logging. The platform starts the Flask app from site/; it does not deploy this VM buffer config.

site/routes/__init__.py registers the page, health, processing API, and upload_refs blueprints. site/routes/main_bp.py serves the page, the picker bundle (/file-picker/...), and raw ES modules (/upload/..., /dist/...). site/routes/health_bp.py exposes /health with the application version.

Main endpoints:

Endpoint	Purpose
GET /	Render the browser UI
GET /health	Liveness and version response
GET /file-picker/<path>	Serve the built picker bundle
GET /upload/<path>, GET /dist/<path>	Serve raw picker ES modules and dist assets
POST /api/upload_refs	Pull browser-uploaded buffer objects into a Flask session
POST /api/upload	Direct multipart upload endpoint
GET /api/session_files/<sid>	Inspect files in a session (diagnostic endpoint; disabled unless ENABLE_SESSION_FILES=1)
GET /api/process_stream/<sid>	Process a session and stream progress using Server-Sent Events
POST /api/process/<sid>	Synchronous/legacy processing endpoint
POST /api/cancel/<sid>	Request a cooperative stop between processing steps
GET /api/download/<sid>	Download the obfuscated ZIP
GET /api/csv/<sid>	Download the mapping CSV separately

Upload bridge and session state

upload/backend/upload_refs/blueprint.py validates names, accepts only URLs under the configured VM-buffer prefix, enforces limits, retries failed pulls, and deletes successfully pulled objects. pull_file.py reads the HTTP response in chunks; the resulting file bytes are then kept in the Flask session.

upload/backend/session/ owns session lifecycle. Sessions use UUID identifiers and process-local RAM; files, the ZIP result, the CSV, and cancellation state are not stored in a database. The configured limits are 50 MiB per file, 500 MiB per session, and a 30-minute TTL. The TTL is paused while processing and resumes afterwards. A Flask process restart loses its sessions and results.

Document-processing core

site/routes/api_bp.py loads all session files and invokes drhider.obfuscate_files once for the set. This lets the core use one shared mapping so the same entity receives the same replacement across documents.

The core package is split by responsibility:

• drhider/extractor.py converts .docx, .pdf, and .doc content to text/Markdown, expands ZIP archives with ZIP-bomb protections, and retains internal CSV extraction/obfuscation code. CSV input is temporarily disabled at upload entry points while reliability work focuses on other formats; the separate output mapping CSV remains enabled. .doc conversion calls the configured conversion service.
• drhider/scanner.py detects entities with regular expressions and LLM-based NER, building the original-to-replacement mapping.
• drhider/replacer.py applies that mapping to extracted text.
• drhider/builder.py builds the obfuscated ZIP and mapping CSV. The mapping CSV is deliberately not included inside the ZIP; it is downloaded through its separate endpoint.
• drhider/llm_client.py calls an OpenAI-compatible LLM endpoint and tracks token/time statistics.

Processing runs in a worker thread for the SSE endpoint. Progress, per-file events, LLM activity, completion, cancellation, and errors are sent to the browser as SSE events. Cancellation is cooperative; the current operation/file may finish before the worker stops.

The LLM client reads LLM_KEY or LLM_API_KEY, with optional LLM_URL (default https://api.aillm.ru/v1/chat/completions) and LLM_MODEL (default gpt-oss-120b), from the Managed Flask environment. LLM_RETRIES sets the number of retries after the first request (default 3), and LLM_RETRY_DELAY sets the base backoff in seconds (default 2.0; delays grow linearly). The .doc converter URL is configured with CONVERT_SERVICE_URL (default https://liberta.containerk8s.services.ngcloud.ru/convert); .doc files are converted sequentially to avoid overloading the conversion service. Secrets belong in service environment settings, never in this repository.

VM213 WebDAV Buffer

The source of the Obfuscator Nginx config is deploy/vm213/nginx/obfuscator-upload.conf. On VM213 it is installed as /etc/nginx/sites-enabled/obfuscator-upload.conf.

• Hostname: obfuscator-upload.sinefo.ru
• WebDAV path: /obfuscator-upload/
• Disk staging directory: /var/www/obfuscator-upload/
• TLS certificate: /etc/letsencrypt/live/obfuscator-upload.sinefo.ru/
• Maximum Nginx request body: 1 GiB; the Flask upload bridge is stricter at 50 MiB per file.
• HTTP redirects to HTTPS; the ACME challenge is served from /var/www/acme.

The CORS allowlist is for the Obfuscator browser origin https://obfuscator.pythonk8s.services.ngcloud.ru. CORS is a browser policy, not authentication; direct HTTP clients are not blocked by CORS. The VM disk is temporary staging, not durable storage. A VM or disk failure can lose files that have not yet been pulled by Flask.

Install a cron job on VM213 to remove orphaned upload-buffer files older than 30 minutes. The cleanup command is:

find /var/www/obfuscator-upload/ -type f -mmin +30 -delete

To update the VM config, update this repository source, install the same file on VM213, run nginx -t, and reload Nginx only if validation succeeds. The Managed Flask deployment and Nginx deployment are independent; deploying one does not deploy the other.

Repository Map

site/
	app.py                 Flask app factory and startup entry point
	routes/                UI, health, and processing/upload API blueprints
	templates/             Browser application page
	static/dist/           Built picker bundles served to the browser
upload/
	frontend/              File picker and browser-to-buffer transport
	backend/session/       In-memory sessions, limits, results, and TTL
	backend/upload_refs/   VM-buffer pull bridge
drhider/
	extractor.py           Document and ZIP input handling
	scanner.py             Regex and LLM entity discovery
	replacer.py            Mapping application
	builder.py             ZIP and mapping CSV output
deploy/vm213/nginx/       Source config for the Obfuscator VM213 buffer