forcloud/obfuscator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Improve LLM and CSV processing

Browse Source 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Repinoid
2026-10-23 10:09:34 +03:00
co-authored by Copilot
parent aa1d15509b
commit 49171c93f9
14 changed files with 315 additions and 92 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+11 -5
View File
@@ -28,7 +28,7 @@ Browser
-> GET /api/csv/<sid> Browser downloads the mapping separately
```
The browser sends files one at a time: it waits for the WebDAV `PUT`, then asks Flask to pull that file before moving to the next one. The buffer therefore holds files only during transit, not for the complete processing session. If the Flask pull fails, the buffer object may remain until manually removed or otherwise cleaned up; the current Nginx config does not define a TTL cleanup job.
The browser sends files one at a time: it waits for the WebDAV `PUT`, then asks Flask to pull that file before moving to the next one. The buffer therefore holds files only during transit, not for the complete processing session. If the Flask pull fails, the buffer object may remain until manually removed or otherwise cleaned up; install the TTL cron cleanup described below on VM213.
## Application Components
@@ -40,7 +40,7 @@ The browser sends files one at a time: it waits for the WebDAV `PUT`, then asks
- `upload/frontend/upload/put_to_vm.js` sends a raw file body with `PUT` and reports progress.
- `upload/frontend/upload/upload_via_vm.js` coordinates the per-file `PUT` and `/api/upload_refs` requests.
The UI accepts `.pdf`, `.doc`, `.docx`, `.txt`, `.md`, and `.zip` inputs. ZIP archives are expanded by the browser picker, and their allowed inner files are uploaded individually. The backend `drhider/extractor.py` also expands ZIPs received directly (for example, through `POST /api/upload`). Per-file (50 MiB) and per-session (500 MiB) limits are enforced by the backend; the picker's entry/size/depth limits apply only while unpacking ZIP archives.
The UI accepts `.pdf`, `.doc`, `.docx`, `.txt`, `.md`, `.csv`, and `.zip` inputs. ZIP archives are expanded by the browser picker, and their allowed inner files are uploaded individually. The backend `drhider/extractor.py` also expands ZIPs received directly (for example, through `POST /api/upload`). Per-file (50 MiB) and per-session (500 MiB) limits are enforced by the backend; the picker's entry/size/depth limits apply only while unpacking ZIP archives.
### Flask composition and routes
@@ -58,7 +58,7 @@ Main endpoints:
| `GET /upload/<path>`, `GET /dist/<path>` | Serve raw picker ES modules and dist assets |
| `POST /api/upload_refs` | Pull browser-uploaded buffer objects into a Flask session |
| `POST /api/upload` | Direct multipart upload endpoint |
| `GET /api/session_files/<sid>` | Inspect files in a session (diagnostic endpoint) |
| `GET /api/session_files/<sid>` | Inspect files in a session (diagnostic endpoint; disabled unless `ENABLE_SESSION_FILES=1`) |
| `GET /api/process_stream/<sid>` | Process a session and stream progress using Server-Sent Events |
| `POST /api/process/<sid>` | Synchronous/legacy processing endpoint |
| `POST /api/cancel/<sid>` | Request a cooperative stop between processing steps |
@@ -77,7 +77,7 @@ Main endpoints:
The core package is split by responsibility:
- `drhider/extractor.py` converts `.docx`, `.pdf`, and `.doc` content to text/Markdown and expands ZIP archives with ZIP-bomb protections. `.doc` conversion calls the configured conversion service.
- `drhider/extractor.py` converts `.docx`, `.pdf`, and `.doc` content to text/Markdown, extracts CSV cell values for scanning, obfuscates CSV cells, and expands ZIP archives with ZIP-bomb protections. `.doc` conversion calls the configured conversion service.
- `drhider/scanner.py` detects entities with regular expressions and LLM-based NER, building the original-to-replacement mapping.
- `drhider/replacer.py` applies that mapping to extracted text.
- `drhider/builder.py` builds the obfuscated ZIP and mapping CSV. The mapping CSV is deliberately not included inside the ZIP; it is downloaded through its separate endpoint.
@@ -85,7 +85,7 @@ The core package is split by responsibility:
Processing runs in a worker thread for the SSE endpoint. Progress, per-file events, LLM activity, completion, cancellation, and errors are sent to the browser as SSE events. Cancellation is cooperative; the current operation/file may finish before the worker stops.
The LLM client reads `LLM_KEY` or `LLM_API_KEY`, with optional `LLM_URL` (default `https://api.aillm.ru/v1/chat/completions`) and `LLM_MODEL` (default `gpt-oss-120b`), from the Managed Flask environment. The `.doc` converter URL is configured with `CONVERT_SERVICE_URL` (default `https://liberta.containerk8s.dev.nubes.ru/convert`); `.doc` files are converted sequentially to avoid overloading the conversion service. Secrets belong in service environment settings, never in this repository.
The LLM client reads `LLM_KEY` or `LLM_API_KEY`, with optional `LLM_URL` (default `https://api.aillm.ru/v1/chat/completions`) and `LLM_MODEL` (default `gpt-oss-120b`), from the Managed Flask environment. `LLM_RETRIES` sets the number of retries after the first request (default `3`), and `LLM_RETRY_DELAY` sets the base backoff in seconds (default `2.0`; delays grow linearly). The `.doc` converter URL is configured with `CONVERT_SERVICE_URL` (default `https://liberta.containerk8s.dev.nubes.ru/convert`); `.doc` files are converted sequentially to avoid overloading the conversion service. Secrets belong in service environment settings, never in this repository.
## VM213 WebDAV Buffer
@@ -100,6 +100,12 @@ The source of the Obfuscator Nginx config is [`deploy/vm213/nginx/obfuscator-upl
The CORS allowlist is for the Obfuscator browser origin `https://obfuscator.pythonk8s.services.ngcloud.ru`. CORS is a browser policy, **not authentication**; direct HTTP clients are not blocked by CORS. The VM disk is temporary staging, not durable storage. A VM or disk failure can lose files that have not yet been pulled by Flask.
Install a cron job on VM213 to remove orphaned upload-buffer files older than 30 minutes. The cleanup command is:
```sh
find /var/www/obfuscator-upload/ -type f -mmin +30 -delete
```
To update the VM config, update this repository source, install the same file on VM213, run `nginx -t`, and reload Nginx only if validation succeeds. The Managed Flask deployment and Nginx deployment are independent; deploying one does not deploy the other.
## Repository Map