Document Obfuscator architecture and VM buffer
This commit is contained in:
Repinoid
@@ -0,0 +1,122 @@
|
||||
# Obfuscator
|
||||
|
||||
Obfuscator accepts documents in a browser, finds sensitive entities, replaces them consistently across a file set, and returns obfuscated documents plus a separate mapping CSV.
|
||||
|
||||
## Runtime Boundary
|
||||
|
||||
The application and its upload buffer are separate deployable components:
|
||||
|
||||
| Component | Runs on | Responsibility |
|
||||
| --- | --- | --- |
|
||||
| Obfuscator Flask application | Managed Flask / Shturval | UI, API, in-memory sessions, document extraction, LLM calls, replacement, and downloads |
|
||||
| Obfuscator upload buffer | VM213, Nginx WebDAV | Temporary staging of browser uploads until Flask pulls them |
|
||||
|
||||
VM213 does not run the Flask application or document-processing code. The buffer is a companion service dedicated to Obfuscator and is not the separate `vmfiles.service` API.
|
||||
|
||||
## Request Flow
|
||||
|
||||
```text
|
||||
Browser
|
||||
-> GET / Flask serves the page
|
||||
-> GET /file-picker/... Flask serves the picker bundle
|
||||
-> PUT obfuscator-upload.sinefo.ru/obfuscator-upload/<key>
|
||||
Nginx writes a temporary file on VM213
|
||||
-> POST /api/upload_refs Flask pulls that file with GET, stores bytes
|
||||
in the session, then sends DELETE to VM213
|
||||
-> GET /api/process_stream/<sid> Flask runs document processing and streams SSE progress
|
||||
-> GET /api/download/<sid> Browser downloads the obfuscated ZIP
|
||||
-> GET /api/csv/<sid> Browser downloads the mapping separately
|
||||
```
|
||||
|
||||
The browser sends files one at a time: it waits for the WebDAV `PUT`, then asks Flask to pull that file before moving to the next one. The buffer therefore holds files only during transit, not for the complete processing session. If the Flask pull fails, the buffer object may remain until manually removed or otherwise cleaned up; the current Nginx config does not define a TTL cleanup job.
|
||||
|
||||
## Application Components
|
||||
|
||||
### Browser UI and picker
|
||||
|
||||
- `site/templates/index.html` provides the Obfuscator UI, sets accepted extensions and limits, and configures the VM buffer URL.
|
||||
- `site/static/dist/file-picker.iife.js` is the browser bundle loaded by the page.
|
||||
- `upload/frontend/index.js` implements the reusable picker API; `upload/frontend/table/` handles file/folder selection, tree rendering, deduplication, and status updates.
|
||||
- `upload/frontend/upload/put_to_vm.js` sends a raw file body with `PUT` and reports progress.
|
||||
- `upload/frontend/upload/upload_via_vm.js` coordinates the per-file `PUT` and `/api/upload_refs` requests.
|
||||
|
||||
The UI accepts `.pdf`, `.doc`, `.docx`, `.txt`, `.md`, and `.zip` inputs. ZIP inputs are expanded by the processing layer. The picker limits a selected set to 1,000 entries, 50 MiB per entry, and 500 MiB total.
|
||||
|
||||
### Flask composition and routes
|
||||
|
||||
`site/app.py` creates the Flask application and configures request limits and logging. The platform starts the Flask app from `site/`; it does not deploy this VM buffer config.
|
||||
|
||||
`site/routes/__init__.py` registers the page, health, processing API, and `upload_refs` blueprints. `site/routes/main_bp.py` serves the page and picker assets. `site/routes/health_bp.py` exposes `/health` with the application version.
|
||||
|
||||
Main endpoints:
|
||||
|
||||
| Endpoint | Purpose |
|
||||
| --- | --- |
|
||||
| `GET /` | Render the browser UI |
|
||||
| `GET /health` | Liveness and version response |
|
||||
| `GET /file-picker/<path>` | Serve the built picker bundle |
|
||||
| `POST /api/upload_refs` | Pull browser-uploaded buffer objects into a Flask session |
|
||||
| `POST /api/upload` | Direct multipart upload endpoint |
|
||||
| `GET /api/session_files/<sid>` | Inspect files in a session (diagnostic endpoint) |
|
||||
| `GET /api/process_stream/<sid>` | Process a session and stream progress using Server-Sent Events |
|
||||
| `POST /api/process/<sid>` | Synchronous/legacy processing endpoint |
|
||||
| `POST /api/cancel/<sid>` | Request a cooperative stop between processing steps |
|
||||
| `GET /api/download/<sid>` | Download the obfuscated ZIP |
|
||||
| `GET /api/csv/<sid>` | Download the mapping CSV separately |
|
||||
|
||||
### Upload bridge and session state
|
||||
|
||||
`upload/backend/upload_refs/blueprint.py` validates names, accepts only URLs under the configured VM-buffer prefix, enforces limits, retries failed pulls, and deletes successfully pulled objects. `pull_file.py` reads the HTTP response in chunks; the resulting file bytes are then kept in the Flask session.
|
||||
|
||||
`upload/backend/session/` owns session lifecycle. Sessions use UUID identifiers and process-local RAM; files, the ZIP result, the CSV, and cancellation state are not stored in a database. The configured limits are 50 MiB per file, 500 MiB per session, and a 30-minute TTL. The TTL is paused while processing and resumes afterwards. A Flask process restart loses its sessions and results.
|
||||
|
||||
### Document-processing core
|
||||
|
||||
`site/routes/api_bp.py` loads all session files and invokes `drhider.obfuscate_files` once for the set. This lets the core use one shared mapping so the same entity receives the same replacement across documents.
|
||||
|
||||
The core package is split by responsibility:
|
||||
|
||||
- `drhider/extractor.py` converts `.docx`, `.pdf`, and `.doc` content to text/Markdown and expands ZIP archives with ZIP-bomb protections. `.doc` conversion calls the configured conversion service.
|
||||
- `drhider/scanner.py` detects entities with regular expressions and LLM-based NER, building the original-to-replacement mapping.
|
||||
- `drhider/replacer.py` applies that mapping to extracted text.
|
||||
- `drhider/builder.py` builds the obfuscated ZIP and mapping CSV. The mapping CSV is deliberately not included inside the ZIP; it is downloaded through its separate endpoint.
|
||||
- `drhider/llm_client.py` calls an OpenAI-compatible LLM endpoint and tracks token/time statistics.
|
||||
|
||||
Processing runs in a worker thread for the SSE endpoint. Progress, per-file events, LLM activity, completion, cancellation, and errors are sent to the browser as SSE events. Cancellation is cooperative; the current operation/file may finish before the worker stops.
|
||||
|
||||
The LLM client reads `LLM_KEY` or `LLM_API_KEY`, with optional `LLM_URL` and `LLM_MODEL`, from the Managed Flask environment. The `.doc` converter URL is configured with `CONVERT_SERVICE_URL`. Secrets belong in service environment settings, never in this repository.
|
||||
|
||||
## VM213 WebDAV Buffer
|
||||
|
||||
The source of the Obfuscator Nginx config is [`deploy/vm213/nginx/obfuscator-upload.conf`](deploy/vm213/nginx/obfuscator-upload.conf). On VM213 it is installed as `/etc/nginx/sites-enabled/obfuscator-upload.conf`.
|
||||
|
||||
- Hostname: `obfuscator-upload.sinefo.ru`
|
||||
- WebDAV path: `/obfuscator-upload/`
|
||||
- Disk staging directory: `/var/www/obfuscator-upload/`
|
||||
- TLS certificate: `/etc/letsencrypt/live/obfuscator-upload.sinefo.ru/`
|
||||
- Maximum Nginx request body: 1 GiB; the Flask upload bridge is stricter at 50 MiB per file.
|
||||
- HTTP redirects to HTTPS; the ACME challenge is served from `/var/www/acme`.
|
||||
|
||||
The CORS allowlist is for the Obfuscator browser origin. CORS is a browser policy, **not authentication**; direct HTTP clients are not blocked by CORS. The VM disk is temporary staging, not durable storage. A VM or disk failure can lose files that have not yet been pulled by Flask.
|
||||
|
||||
To update the VM config, update this repository source, install the same file on VM213, run `nginx -t`, and reload Nginx only if validation succeeds. The Managed Flask deployment and Nginx deployment are independent; deploying one does not deploy the other.
|
||||
|
||||
## Repository Map
|
||||
|
||||
```text
|
||||
site/
|
||||
app.py Flask app factory and startup entry point
|
||||
routes/ UI, health, and processing/upload API blueprints
|
||||
templates/ Browser application page
|
||||
static/dist/ Built picker bundles served to the browser
|
||||
upload/
|
||||
frontend/ File picker and browser-to-buffer transport
|
||||
backend/session/ In-memory sessions, limits, results, and TTL
|
||||
backend/upload_refs/ VM-buffer pull bridge
|
||||
drhider/
|
||||
extractor.py Document and ZIP input handling
|
||||
scanner.py Regex and LLM entity discovery
|
||||
replacer.py Mapping application
|
||||
builder.py ZIP and mapping CSV output
|
||||
deploy/vm213/nginx/ Source config for the Obfuscator VM213 buffer
|
||||
```
|
||||
@@ -0,0 +1,58 @@
|
||||
# Obfuscator's temporary upload buffer on VM213. This is Nginx/WebDAV config,
|
||||
# not the Managed Flask application or document-processing code.
|
||||
# Install as /etc/nginx/sites-enabled/obfuscator-upload.conf.
|
||||
# The sites-enabled wildcard is included from Nginx's http context, where map is valid.
|
||||
# Keep this file in sync with the installed VM config.
|
||||
|
||||
# Browser CORS allowlist for the Obfuscator Managed Flask origin.
|
||||
# CORS is a browser policy, not authentication for direct HTTP clients.
|
||||
map $http_origin $obfuscator_upload_cors_origin {
|
||||
default "";
|
||||
~^https://obfuscator\.pythonk8s\.services\.ngcloud\.ru$ $http_origin;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name obfuscator-upload.sinefo.ru;
|
||||
|
||||
# Keep the ACME HTTP-01 challenge available for certificate renewal.
|
||||
location /.well-known/acme-challenge/ {
|
||||
root /var/www/acme;
|
||||
}
|
||||
|
||||
# Send all other HTTP traffic to TLS.
|
||||
location / {
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name obfuscator-upload.sinefo.ru;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/obfuscator-upload.sinefo.ru/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/obfuscator-upload.sinefo.ru/privkey.pem;
|
||||
include /etc/letsencrypt/options-ssl-nginx.conf;
|
||||
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
|
||||
|
||||
client_max_body_size 1024m;
|
||||
|
||||
# Temporary WebDAV staging directory: browser PUT, Flask GET/pull, then DELETE.
|
||||
location /obfuscator-upload/ {
|
||||
alias /var/www/obfuscator-upload/;
|
||||
dav_methods PUT DELETE;
|
||||
create_full_put_path on;
|
||||
client_max_body_size 1024m;
|
||||
add_header Access-Control-Allow-Origin $obfuscator_upload_cors_origin always;
|
||||
add_header Access-Control-Allow-Methods "PUT, GET, OPTIONS, DELETE" always;
|
||||
add_header Access-Control-Allow-Headers "Content-Type" always;
|
||||
add_header Access-Control-Max-Age 86400 always;
|
||||
if ($request_method = OPTIONS) { return 204; }
|
||||
}
|
||||
|
||||
# Do not serve paths outside the upload endpoint.
|
||||
location / {
|
||||
return 404;
|
||||
}
|
||||
}
|
||||
# End of Obfuscator's VM213 upload-buffer config.
|
||||
+1
-1
@@ -21,7 +21,7 @@ if _sys_path_root not in sys.path:
|
||||
sys.path.insert(0, _sys_path_root)
|
||||
|
||||
# Версия приложения (меняется при изменениях)
|
||||
VERSION = "1.0.1"
|
||||
VERSION = "1.0.2"
|
||||
|
||||
|
||||
def setup_logging():
|
||||
|
||||
Reference in New Issue
Block a user