Improve LLM and CSV processing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Repinoid
@@ -28,7 +28,7 @@ Browser
|
||||
-> GET /api/csv/<sid> Browser downloads the mapping separately
|
||||
```
|
||||
|
||||
The browser sends files one at a time: it waits for the WebDAV `PUT`, then asks Flask to pull that file before moving to the next one. The buffer therefore holds files only during transit, not for the complete processing session. If the Flask pull fails, the buffer object may remain until manually removed or otherwise cleaned up; the current Nginx config does not define a TTL cleanup job.
|
||||
The browser sends files one at a time: it waits for the WebDAV `PUT`, then asks Flask to pull that file before moving to the next one. The buffer therefore holds files only during transit, not for the complete processing session. If the Flask pull fails, the buffer object may remain until manually removed or otherwise cleaned up; install the TTL cron cleanup described below on VM213.
|
||||
|
||||
## Application Components
|
||||
|
||||
@@ -40,7 +40,7 @@ The browser sends files one at a time: it waits for the WebDAV `PUT`, then asks
|
||||
- `upload/frontend/upload/put_to_vm.js` sends a raw file body with `PUT` and reports progress.
|
||||
- `upload/frontend/upload/upload_via_vm.js` coordinates the per-file `PUT` and `/api/upload_refs` requests.
|
||||
|
||||
The UI accepts `.pdf`, `.doc`, `.docx`, `.txt`, `.md`, and `.zip` inputs. ZIP archives are expanded by the browser picker, and their allowed inner files are uploaded individually. The backend `drhider/extractor.py` also expands ZIPs received directly (for example, through `POST /api/upload`). Per-file (50 MiB) and per-session (500 MiB) limits are enforced by the backend; the picker's entry/size/depth limits apply only while unpacking ZIP archives.
|
||||
The UI accepts `.pdf`, `.doc`, `.docx`, `.txt`, `.md`, `.csv`, and `.zip` inputs. ZIP archives are expanded by the browser picker, and their allowed inner files are uploaded individually. The backend `drhider/extractor.py` also expands ZIPs received directly (for example, through `POST /api/upload`). Per-file (50 MiB) and per-session (500 MiB) limits are enforced by the backend; the picker's entry/size/depth limits apply only while unpacking ZIP archives.
|
||||
|
||||
### Flask composition and routes
|
||||
|
||||
@@ -58,7 +58,7 @@ Main endpoints:
|
||||
| `GET /upload/<path>`, `GET /dist/<path>` | Serve raw picker ES modules and dist assets |
|
||||
| `POST /api/upload_refs` | Pull browser-uploaded buffer objects into a Flask session |
|
||||
| `POST /api/upload` | Direct multipart upload endpoint |
|
||||
| `GET /api/session_files/<sid>` | Inspect files in a session (diagnostic endpoint) |
|
||||
| `GET /api/session_files/<sid>` | Inspect files in a session (diagnostic endpoint; disabled unless `ENABLE_SESSION_FILES=1`) |
|
||||
| `GET /api/process_stream/<sid>` | Process a session and stream progress using Server-Sent Events |
|
||||
| `POST /api/process/<sid>` | Synchronous/legacy processing endpoint |
|
||||
| `POST /api/cancel/<sid>` | Request a cooperative stop between processing steps |
|
||||
@@ -77,7 +77,7 @@ Main endpoints:
|
||||
|
||||
The core package is split by responsibility:
|
||||
|
||||
- `drhider/extractor.py` converts `.docx`, `.pdf`, and `.doc` content to text/Markdown and expands ZIP archives with ZIP-bomb protections. `.doc` conversion calls the configured conversion service.
|
||||
- `drhider/extractor.py` converts `.docx`, `.pdf`, and `.doc` content to text/Markdown, extracts CSV cell values for scanning, obfuscates CSV cells, and expands ZIP archives with ZIP-bomb protections. `.doc` conversion calls the configured conversion service.
|
||||
- `drhider/scanner.py` detects entities with regular expressions and LLM-based NER, building the original-to-replacement mapping.
|
||||
- `drhider/replacer.py` applies that mapping to extracted text.
|
||||
- `drhider/builder.py` builds the obfuscated ZIP and mapping CSV. The mapping CSV is deliberately not included inside the ZIP; it is downloaded through its separate endpoint.
|
||||
@@ -85,7 +85,7 @@ The core package is split by responsibility:
|
||||
|
||||
Processing runs in a worker thread for the SSE endpoint. Progress, per-file events, LLM activity, completion, cancellation, and errors are sent to the browser as SSE events. Cancellation is cooperative; the current operation/file may finish before the worker stops.
|
||||
|
||||
The LLM client reads `LLM_KEY` or `LLM_API_KEY`, with optional `LLM_URL` (default `https://api.aillm.ru/v1/chat/completions`) and `LLM_MODEL` (default `gpt-oss-120b`), from the Managed Flask environment. The `.doc` converter URL is configured with `CONVERT_SERVICE_URL` (default `https://liberta.containerk8s.dev.nubes.ru/convert`); `.doc` files are converted sequentially to avoid overloading the conversion service. Secrets belong in service environment settings, never in this repository.
|
||||
The LLM client reads `LLM_KEY` or `LLM_API_KEY`, with optional `LLM_URL` (default `https://api.aillm.ru/v1/chat/completions`) and `LLM_MODEL` (default `gpt-oss-120b`), from the Managed Flask environment. `LLM_RETRIES` sets the number of retries after the first request (default `3`), and `LLM_RETRY_DELAY` sets the base backoff in seconds (default `2.0`; delays grow linearly). The `.doc` converter URL is configured with `CONVERT_SERVICE_URL` (default `https://liberta.containerk8s.dev.nubes.ru/convert`); `.doc` files are converted sequentially to avoid overloading the conversion service. Secrets belong in service environment settings, never in this repository.
|
||||
|
||||
## VM213 WebDAV Buffer
|
||||
|
||||
@@ -100,6 +100,12 @@ The source of the Obfuscator Nginx config is [`deploy/vm213/nginx/obfuscator-upl
|
||||
|
||||
The CORS allowlist is for the Obfuscator browser origin `https://obfuscator.pythonk8s.services.ngcloud.ru`. CORS is a browser policy, **not authentication**; direct HTTP clients are not blocked by CORS. The VM disk is temporary staging, not durable storage. A VM or disk failure can lose files that have not yet been pulled by Flask.
|
||||
|
||||
Install a cron job on VM213 to remove orphaned upload-buffer files older than 30 minutes. The cleanup command is:
|
||||
|
||||
```sh
|
||||
find /var/www/obfuscator-upload/ -type f -mmin +30 -delete
|
||||
```
|
||||
|
||||
To update the VM config, update this repository source, install the same file on VM213, run `nginx -t`, and reload Nginx only if validation succeeds. The Managed Flask deployment and Nginx deployment are independent; deploying one does not deploy the other.
|
||||
|
||||
## Repository Map
|
||||
|
||||
@@ -55,4 +55,8 @@ server {
|
||||
return 404;
|
||||
}
|
||||
}
|
||||
# End of Obfuscator's VM213 upload-buffer config.
|
||||
# End of Obfuscator's VM213 upload-buffer config.
|
||||
|
||||
# TTL cleanup (configure separately as a cron job on VM213; this file does not
|
||||
# install or schedule the job):
|
||||
# find /var/www/obfuscator-upload/ -type f -mmin +30 -delete
|
||||
+2
-16
@@ -8,13 +8,13 @@
|
||||
|
||||
import io
|
||||
import csv
|
||||
import os
|
||||
import zipfile
|
||||
import re
|
||||
import time
|
||||
from typing import Dict, List, Tuple
|
||||
|
||||
from .config import ENTITY_PATTERNS, COMPANY_PATTERN
|
||||
from .names import resolve_name
|
||||
|
||||
|
||||
def build_zip(files: List[Tuple[str, bytes]], mapping_csv: str = "") -> bytes:
|
||||
@@ -39,24 +39,10 @@ def build_zip(files: List[Tuple[str, bytes]], mapping_csv: str = "") -> bytes:
|
||||
# Дедупликация имён: имя+одинаковый контент → пропуск; имя+разный контент → суффикс
|
||||
seen: Dict[str, bytes] = {}
|
||||
|
||||
def _resolve_name(name: str, content: bytes) -> str:
|
||||
if name not in seen:
|
||||
seen[name] = content
|
||||
return name
|
||||
if seen[name] == content:
|
||||
return None # точный дубль — пропускаем
|
||||
base, ext = os.path.splitext(name)
|
||||
n = 2
|
||||
while f"{base}_{n}{ext}" in seen:
|
||||
n += 1
|
||||
resolved = f"{base}_{n}{ext}"
|
||||
seen[resolved] = content
|
||||
return resolved
|
||||
|
||||
with zipfile.ZipFile(buf, 'w', zipfile.ZIP_DEFLATED) as zf:
|
||||
# Добавляем обфусцированные файлы
|
||||
for fname, content in files:
|
||||
out_name = _resolve_name(fname, content)
|
||||
out_name = resolve_name(fname, content, seen)
|
||||
if out_name is None:
|
||||
continue
|
||||
info = zipfile.ZipInfo(out_name)
|
||||
|
||||
+87
-8
@@ -1,10 +1,11 @@
|
||||
"""
|
||||
Извлечение текста из документов — все форматы → Markdown.
|
||||
Извлечение текста из документов и подготовка CSV к обфускации.
|
||||
|
||||
Поддерживает:
|
||||
- .docx → MD (python-docx: стили, жирный/курсив, таблицы)
|
||||
- .pdf → MD (pdfplumber: текст + таблицы, без форматирования)
|
||||
- .doc → MD (LibreOffice headless: .doc → .docx → штатный docx_to_markdown)
|
||||
- .doc → HTTP-сервис конвертации (CONVERT_SERVICE_URL) → .docx → Markdown
|
||||
- .csv → текст ячеек для сканирования и обезличенный CSV на выходе
|
||||
- .txt и прочие → как есть
|
||||
|
||||
Также содержит:
|
||||
@@ -12,13 +13,16 @@
|
||||
"""
|
||||
|
||||
import io
|
||||
import csv
|
||||
import os
|
||||
import zipfile
|
||||
import tempfile
|
||||
import shutil
|
||||
import subprocess
|
||||
import logging
|
||||
from typing import List, Tuple
|
||||
from typing import Dict, List, Tuple
|
||||
|
||||
from . import replacer
|
||||
|
||||
log = logging.getLogger("drhider")
|
||||
|
||||
@@ -199,6 +203,77 @@ def doc_to_markdown(content: bytes) -> str:
|
||||
# Единая точка входа
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def decode_text(content: bytes) -> str:
|
||||
"""Декодировать текст как UTF-8, затем CP1251, с Latin-1 последним fallback."""
|
||||
try:
|
||||
return content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
# Блокнот и выгрузки 1С часто сохраняют кириллицу в CP1251.
|
||||
try:
|
||||
return content.decode("cp1251")
|
||||
except UnicodeDecodeError:
|
||||
return content.decode("latin-1", errors="replace")
|
||||
|
||||
|
||||
def detect_delimiter(text: str) -> str:
|
||||
"""Определить CSV-разделитель, используя Sniffer и детерминированный fallback."""
|
||||
delimiters = [",", ";", "\t"]
|
||||
nonempty_lines = [line for line in text.splitlines() if line.strip()]
|
||||
if nonempty_lines:
|
||||
sample = "\n".join(nonempty_lines[:10])
|
||||
try:
|
||||
return csv.Sniffer().sniff(sample, delimiters="".join(delimiters)).delimiter
|
||||
except csv.Error:
|
||||
first_line = nonempty_lines[0]
|
||||
return max(delimiters, key=first_line.count)
|
||||
return ","
|
||||
|
||||
|
||||
def csv_to_text(content: bytes) -> str:
|
||||
"""Распарсить CSV и склеить значения ячеек построчно для сканирования."""
|
||||
text = decode_text(content).removeprefix("\ufeff")
|
||||
delimiter = detect_delimiter(text)
|
||||
reader = csv.reader(
|
||||
io.StringIO(text, newline=""),
|
||||
delimiter=delimiter,
|
||||
quotechar='"',
|
||||
strict=True,
|
||||
)
|
||||
# Каждая ячейка отделена переводом строки, чтобы сущности из соседних ячеек
|
||||
# не объединялись в строку, по которой работают regex и LLM.
|
||||
return "\n".join(cell for row in reader for cell in row)
|
||||
|
||||
|
||||
def obfuscate_csv(content: bytes, mapping: Dict[str, str],
|
||||
sorted_keys: List[str], compiled_re) -> bytes:
|
||||
"""Заменить сущности в CSV-ячейках, сохранив порядок строк и столбцов."""
|
||||
text = decode_text(content).removeprefix("\ufeff")
|
||||
delimiter = detect_delimiter(text)
|
||||
reader = csv.reader(
|
||||
io.StringIO(text, newline=""),
|
||||
delimiter=delimiter,
|
||||
quotechar='"',
|
||||
strict=True,
|
||||
)
|
||||
output = io.StringIO(newline="")
|
||||
writer = csv.writer(
|
||||
output,
|
||||
delimiter=delimiter,
|
||||
quotechar='"',
|
||||
lineterminator="\n",
|
||||
strict=True,
|
||||
)
|
||||
for row in reader:
|
||||
writer.writerow([
|
||||
replacer.apply_replacements(cell, mapping, sorted_keys, compiled_re)
|
||||
for cell in row
|
||||
])
|
||||
|
||||
# csv.writer нормализует кавычки/переводы строк; побайтовое сохранение
|
||||
# исходного формата не является целью. BOM помогает Excel открыть кириллицу.
|
||||
return ("\ufeff" + output.getvalue()).encode("utf-8")
|
||||
|
||||
|
||||
def extract_text(fname: str, content: bytes, ctype: str = "") -> str:
|
||||
"""Извлечь текст из одного файла → Markdown.
|
||||
|
||||
@@ -217,22 +292,26 @@ def extract_text(fname: str, content: bytes, ctype: str = "") -> str:
|
||||
try:
|
||||
return docx_to_markdown(content)
|
||||
except ImportError:
|
||||
return content.decode('utf-8', errors='replace')
|
||||
return decode_text(content)
|
||||
|
||||
# ── .pdf: текст + таблицы ──
|
||||
elif ext == '.pdf':
|
||||
try:
|
||||
return pdf_to_markdown(content)
|
||||
except ImportError:
|
||||
return content.decode('utf-8', errors='replace')
|
||||
return decode_text(content)
|
||||
|
||||
# ── .doc: бинарный → LibreOffice → DOCX → Markdown ──
|
||||
# ── .doc: бинарный → HTTP-сервис конвертации → DOCX → Markdown ──
|
||||
elif ext == '.doc':
|
||||
return doc_to_markdown(content)
|
||||
|
||||
# ── .csv: ячейки для обнаружения сущностей ──
|
||||
elif ext == '.csv':
|
||||
return csv_to_text(content)
|
||||
|
||||
# ── .txt и прочие ──
|
||||
else:
|
||||
return content.decode('utf-8', errors='replace')
|
||||
return decode_text(content)
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
@@ -338,7 +417,7 @@ def expand_zips(files: List[Tuple[str, bytes, str]]) -> List[Tuple[str, bytes, s
|
||||
continue
|
||||
# Синхронизация с фронтом (listZipFiles): из zip берём ТОЛЬКО документы
|
||||
low = name.lower()
|
||||
if not (low.endswith('.zip') or low.endswith(('.pdf', '.doc', '.docx', '.txt', '.md'))):
|
||||
if not (low.endswith('.zip') or low.endswith(('.pdf', '.doc', '.docx', '.txt', '.md', '.csv'))):
|
||||
continue
|
||||
|
||||
inner_data = zf.read(info)
|
||||
|
||||
+39
-6
@@ -11,6 +11,8 @@ LLM-клиент для Obfuscator.
|
||||
|
||||
import os
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
@@ -21,9 +23,12 @@ class LLMClient:
|
||||
LLM_API_KEY или LLM_KEY — ключ API
|
||||
LLM_URL — URL эндпоинта (по умолчанию https://api.aillm.ru/v1/chat/completions)
|
||||
LLM_MODEL — модель (по умолчанию gpt-oss-120b)
|
||||
LLM_RETRIES — число повторов после первой попытки (по умолчанию 3)
|
||||
LLM_RETRY_DELAY — базовая задержка backoff в секундах (по умолчанию 2)
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
def __init__(self, retries: Optional[int] = None,
|
||||
retry_delay: Optional[float] = None):
|
||||
"""Инициализировать клиент. httpx импортируется лениво.
|
||||
|
||||
Ведёт кумулятивные счётчики за всё время жизни клиента:
|
||||
@@ -36,6 +41,13 @@ class LLMClient:
|
||||
llm_started — time.time() начала текущего LLM-вызова
|
||||
"""
|
||||
self._httpx = httpx
|
||||
self.retries = int(os.environ.get("LLM_RETRIES", "3")) if retries is None else retries
|
||||
self.retry_delay = (
|
||||
float(os.environ.get("LLM_RETRY_DELAY", "2.0"))
|
||||
if retry_delay is None else retry_delay
|
||||
)
|
||||
if self.retries < 0 or self.retry_delay < 0:
|
||||
raise ValueError("LLM retries and retry delay must be non-negative")
|
||||
self.tokens_prompt = 0
|
||||
self.tokens_completion = 0
|
||||
self.llm_sec = 0.0
|
||||
@@ -52,6 +64,31 @@ class LLMClient:
|
||||
return time.time() - self.llm_started
|
||||
return 0.0
|
||||
|
||||
def _post_with_retries(self, url: str, json: dict, headers: dict,
|
||||
timeout: int) -> httpx.Response:
|
||||
"""Повторить только временные транспортные ошибки и выбранные HTTP-статусы."""
|
||||
for attempt in range(self.retries + 1):
|
||||
try:
|
||||
response = self._httpx.post(
|
||||
url, json=json, headers=headers, timeout=timeout
|
||||
)
|
||||
self.llm_sec += response.elapsed.total_seconds()
|
||||
response.raise_for_status()
|
||||
return response
|
||||
except (self._httpx.TimeoutException, self._httpx.TransportError):
|
||||
if attempt >= self.retries:
|
||||
raise
|
||||
except self._httpx.HTTPStatusError as e:
|
||||
# Retry only throttling and server failures; 400/401/403 require
|
||||
# a corrected request or credentials and will not improve on retry.
|
||||
if e.response.status_code not in (429, 500, 502, 503, 504):
|
||||
raise
|
||||
if attempt >= self.retries:
|
||||
raise
|
||||
|
||||
# 120 секунд на попытку и backoff означают около 8 минут в худшем случае.
|
||||
time.sleep(self.retry_delay * (attempt + 1))
|
||||
|
||||
def complete(self, prompt: str) -> str:
|
||||
"""Отправить промпт в LLM и вернуть текст ответа.
|
||||
|
||||
@@ -76,7 +113,7 @@ class LLMClient:
|
||||
self.llm_active = True
|
||||
self.llm_started = time.time()
|
||||
try:
|
||||
r = self._httpx.post(
|
||||
r = self._post_with_retries(
|
||||
url,
|
||||
json={
|
||||
"model": model,
|
||||
@@ -90,10 +127,6 @@ class LLMClient:
|
||||
},
|
||||
timeout=120,
|
||||
)
|
||||
r.raise_for_status()
|
||||
|
||||
# Время запроса (реальное время HTTP-вызова LLM)
|
||||
self.llm_sec += r.elapsed.total_seconds()
|
||||
|
||||
data = r.json()
|
||||
# Счётчики токенов из usage (могут отсутствовать у некоторых прокси)
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
"""Helpers for assigning stable, collision-free output filenames."""
|
||||
|
||||
import os
|
||||
from typing import Dict, Optional
|
||||
|
||||
|
||||
def resolve_name(name: str, content: bytes, seen: Dict[str, bytes]) -> Optional[str]:
|
||||
"""Return a unique name, or None when the same name and content were seen.
|
||||
|
||||
Name collisions with different content receive numeric suffixes before
|
||||
the extension, while exact duplicates are omitted.
|
||||
"""
|
||||
if name not in seen:
|
||||
seen[name] = content
|
||||
return name
|
||||
if seen[name] == content:
|
||||
return None
|
||||
|
||||
base, ext = os.path.splitext(name)
|
||||
index = 2
|
||||
while f"{base}_{index}{ext}" in seen:
|
||||
index += 1
|
||||
resolved = f"{base}_{index}{ext}"
|
||||
seen[resolved] = content
|
||||
return resolved
|
||||
+31
-14
@@ -17,10 +17,19 @@ from . import extractor
|
||||
from . import scanner
|
||||
from . import replacer
|
||||
from . import builder
|
||||
from .names import resolve_name
|
||||
|
||||
log = logging.getLogger("drhider")
|
||||
|
||||
|
||||
class LLMIncompleteError(Exception):
|
||||
"""Processing cannot safely return files when one or more LLM chunks failed."""
|
||||
|
||||
def __init__(self, llm_errors: int):
|
||||
self.llm_errors = llm_errors
|
||||
super().__init__("LLM processing incomplete")
|
||||
|
||||
|
||||
def _dedupe_file_names(
|
||||
files: List[Tuple[str, bytes, str]]
|
||||
) -> List[Tuple[str, bytes, str]]:
|
||||
@@ -39,17 +48,9 @@ def _dedupe_file_names(
|
||||
out: List[Tuple[str, bytes, str]] = []
|
||||
|
||||
for fname, content, ctype in files:
|
||||
candidate = fname
|
||||
if candidate in seen:
|
||||
if seen[candidate] == content:
|
||||
continue # точный дубль — пропускаем
|
||||
base, ext = os.path.splitext(fname)
|
||||
n = 2
|
||||
while f"{base}_{n}{ext}" in seen:
|
||||
n += 1
|
||||
candidate = f"{base}_{n}{ext}"
|
||||
seen[candidate] = content
|
||||
out.append((candidate, content, ctype))
|
||||
candidate = resolve_name(fname, content, seen)
|
||||
if candidate is not None:
|
||||
out.append((candidate, content, ctype))
|
||||
|
||||
return out
|
||||
|
||||
@@ -177,13 +178,18 @@ class TwoPassObfuscator:
|
||||
cancelled_llm = cancelled_extract
|
||||
if not cancelled_llm and self._llm_client:
|
||||
try:
|
||||
scanner.scan_llm_ner(
|
||||
llm_failures = scanner.scan_llm_ner(
|
||||
all_texts, self._mapping, self._llm_client, self._counters,
|
||||
cancel_event=cancel_event, file_progress=_fp,
|
||||
)
|
||||
except scanner.CancelRequested:
|
||||
except scanner.CancelRequested as e:
|
||||
if e.llm_failures:
|
||||
raise LLMIncompleteError(e.llm_failures)
|
||||
log.info("obfuscate: LLM cancelled, llm_done=%d", len(llm_done))
|
||||
cancelled_llm = True
|
||||
else:
|
||||
if llm_failures:
|
||||
raise LLMIncompleteError(llm_failures)
|
||||
|
||||
# Предсортировать ключи один раз (по убыванию длины)
|
||||
self._sorted_keys = sorted(
|
||||
@@ -220,7 +226,18 @@ class TwoPassObfuscator:
|
||||
t0 = time.time()
|
||||
obf_content = content # По умолчанию — без изменений
|
||||
|
||||
if fname.endswith('.doc'):
|
||||
if fname.lower().endswith('.csv'):
|
||||
try:
|
||||
obf_content = extractor.obfuscate_csv(
|
||||
content, self._mapping, self._sorted_keys, self._compiled_re
|
||||
)
|
||||
except Exception as e:
|
||||
log.warning("obfuscate: skip CSV %r during replacement: %r", fname, e)
|
||||
skipped.add(fname)
|
||||
if progress_cb:
|
||||
progress_cb("done", i, total, display_name, 0.0)
|
||||
continue
|
||||
elif fname.endswith('.doc'):
|
||||
# .doc — конвертируется через liberta, обрабатывается как .docx
|
||||
md_text = all_texts.get(fname, '')
|
||||
replaced = replacer.apply_replacements(
|
||||
|
||||
+61
-18
@@ -115,6 +115,9 @@ def scan_regex(text: str, mapping: Dict[str, str], counters: Dict[str, int]) ->
|
||||
class CancelRequested(Exception):
|
||||
"""Обработка прервана пользователем (мягкая остановка между файлами LLM)."""
|
||||
|
||||
def __init__(self, llm_failures: int = 0):
|
||||
self.llm_failures = llm_failures
|
||||
|
||||
# Приоритет границ для чанков (от предпочтительных к жёстким)
|
||||
_CHUNK_BOUNDARIES = ['\n\n', '\n', '. ', '? ', '! ', '; ', ', ', ' ']
|
||||
_CHUNK_SIZE = 6000 # символов на чанк (≈1500-2000 токенов RU — NER-качество)
|
||||
@@ -202,26 +205,53 @@ def _build_llm_prompt(text: str, already_found: List[str]) -> str:
|
||||
|
||||
|
||||
def _call_llm(text: str, llm_client) -> List[dict]:
|
||||
"""Один вызов LLM по фрагменту → список сущностей (пустой при ошибке)."""
|
||||
"""Один вызов LLM по фрагменту → проверенный список сущностей."""
|
||||
raw = llm_client.complete(text).strip()
|
||||
if raw.startswith("```"):
|
||||
lines = raw.splitlines()
|
||||
if lines and lines[0].startswith("```"):
|
||||
lines = lines[1:]
|
||||
if lines and lines[-1].strip() == "```":
|
||||
lines = lines[:-1]
|
||||
raw = "\n".join(lines).strip()
|
||||
|
||||
try:
|
||||
raw = llm_client.complete(text)
|
||||
raw = raw.strip()
|
||||
if raw.startswith("```"):
|
||||
raw = raw.split("\n", 1)[1]
|
||||
if raw.endswith("```"):
|
||||
raw = raw[:-3]
|
||||
entities = json.loads(raw)
|
||||
if not isinstance(entities, list):
|
||||
return []
|
||||
return entities
|
||||
except Exception as e:
|
||||
log.warning("LLM NER failed: %s", e)
|
||||
return []
|
||||
except json.JSONDecodeError as original_error:
|
||||
# Модель иногда добавляет текст вокруг массива; raw_decode корректно
|
||||
# учитывает вложенные массивы и квадратные скобки внутри JSON-строк.
|
||||
decoder = json.JSONDecoder()
|
||||
entities = None
|
||||
for index, char in enumerate(raw):
|
||||
if char != "[":
|
||||
continue
|
||||
try:
|
||||
candidate, _ = decoder.raw_decode(raw, index)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(candidate, list):
|
||||
entities = candidate
|
||||
break
|
||||
if entities is None:
|
||||
raise ValueError("LLM response does not contain a JSON array") from original_error
|
||||
|
||||
if not isinstance(entities, list):
|
||||
raise ValueError("LLM response must be a JSON array")
|
||||
for entity in entities:
|
||||
if not isinstance(entity, dict):
|
||||
raise ValueError("LLM entity must be a JSON object")
|
||||
value = entity.get("value")
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise ValueError("LLM entity value must be a non-empty string")
|
||||
entity_type = entity.get("type")
|
||||
if entity_type is not None and not isinstance(entity_type, str):
|
||||
raise ValueError("LLM entity type must be a string")
|
||||
return entities
|
||||
|
||||
|
||||
def scan_llm_ner(all_texts: Dict[str, str], mapping: Dict[str, str],
|
||||
llm_client, counters: Dict[str, int],
|
||||
cancel_event=None, file_progress=None) -> None:
|
||||
cancel_event=None, file_progress=None) -> int:
|
||||
"""Универсальное LLM-обнаружение приватных данных (пофайлово, целиком).
|
||||
|
||||
Каждый файл обрабатывается ПОЛНОСТЬЮ: текст разбивается на чанки
|
||||
@@ -243,11 +273,12 @@ def scan_llm_ner(all_texts: Dict[str, str], mapping: Dict[str, str],
|
||||
file_items = sorted(
|
||||
all_texts.items(), key=lambda kv: len(kv[1]), reverse=True
|
||||
)
|
||||
llm_failures = 0
|
||||
|
||||
for fname, full_text in file_items:
|
||||
# Мягкая остановка ТОЛЬКО между файлами — текущий файл добирается до конца
|
||||
if cancel_event is not None and cancel_event.is_set():
|
||||
raise CancelRequested()
|
||||
raise CancelRequested(llm_failures)
|
||||
|
||||
if not full_text or full_text.startswith("[DOC binary"):
|
||||
continue
|
||||
@@ -265,11 +296,19 @@ def scan_llm_ner(all_texts: Dict[str, str], mapping: Dict[str, str],
|
||||
for c in chunks]
|
||||
all_ent = []
|
||||
for fut in futures:
|
||||
all_ent.extend(fut.result())
|
||||
try:
|
||||
all_ent.extend(fut.result())
|
||||
except Exception as e:
|
||||
llm_failures += 1
|
||||
log.warning("LLM NER chunk failed in %r: %s", fname, e)
|
||||
else:
|
||||
all_ent = []
|
||||
for k, c in enumerate(chunks):
|
||||
all_ent.extend(_call_llm(_build_llm_prompt(c, mapping.keys()), llm_client))
|
||||
try:
|
||||
all_ent.extend(_call_llm(_build_llm_prompt(c, mapping.keys()), llm_client))
|
||||
except Exception as e:
|
||||
llm_failures += 1
|
||||
log.warning("LLM NER chunk failed in %r: %s", fname, e)
|
||||
if file_progress:
|
||||
file_progress("file_chunk", fname,
|
||||
chunks_done=k + 1, chunks_total=len(chunks))
|
||||
@@ -287,8 +326,12 @@ def scan_llm_ner(all_texts: Dict[str, str], mapping: Dict[str, str],
|
||||
if not _verify_entity(val, full_text):
|
||||
log.debug("LLM hallucination dropped: %r", val)
|
||||
continue
|
||||
ent_type = ent.get("type", "").strip().lower().replace(" ", "_")
|
||||
raw_type = ent.get("type")
|
||||
ent_type = raw_type.strip().lower().replace(" ", "_") if raw_type else ""
|
||||
ent_type = ent_type or "other_id"
|
||||
mapping[val] = _next_token(ent_type, counters)
|
||||
|
||||
if file_progress:
|
||||
file_progress("file_done", fname, elapsed=round(time.time() - t0, 2))
|
||||
|
||||
return llm_failures
|
||||
|
||||
+2
-3
@@ -22,7 +22,7 @@ if _sys_path_root not in sys.path:
|
||||
sys.path.insert(0, _sys_path_root)
|
||||
|
||||
# Версия приложения (меняется при изменениях)
|
||||
VERSION = "1.0.2"
|
||||
VERSION = "1.0.3"
|
||||
|
||||
|
||||
def setup_logging():
|
||||
@@ -73,6 +73,7 @@ def create_app():
|
||||
# Конфигурация
|
||||
app.config["VERSION"] = VERSION
|
||||
app.config["MAX_CONTENT_LENGTH"] = 50 * 1024 * 1024 # 50 MB (лимит на один файл)
|
||||
app.config["ENABLE_SESSION_FILES"] = os.environ.get("ENABLE_SESSION_FILES", "0") == "1"
|
||||
|
||||
# Регистрируем blueprint'ы
|
||||
from routes import register_routes
|
||||
@@ -98,5 +99,3 @@ if __name__ == "__main__":
|
||||
from waitress import serve
|
||||
serve(app, host="0.0.0.0", port=5000, threads=8)
|
||||
|
||||
|
||||
|
||||
|
||||
+36
-12
@@ -17,12 +17,13 @@ import queue
|
||||
import threading
|
||||
import time
|
||||
import zipfile
|
||||
import traceback
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from flask import Blueprint, request, send_file, jsonify, Response, stream_with_context
|
||||
from flask import (Blueprint, current_app, request, send_file, jsonify, Response,
|
||||
stream_with_context)
|
||||
|
||||
from drhider import obfuscate_files, LLMClient
|
||||
from drhider.obfuscator import LLMIncompleteError
|
||||
from upload.backend.session import (create_session, add_file, get_files, store_result,
|
||||
get_result, store_csv, get_csv, cleanup, file_count,
|
||||
MAX_FILE_BYTES, pause_ttl, resume_ttl,
|
||||
@@ -78,6 +79,8 @@ def upload():
|
||||
@api_bp.route("/session_files/<sid>", methods=["GET"])
|
||||
def session_files(sid):
|
||||
"""Отладка: список файлов сессии с размерами (для теста загрузки)."""
|
||||
if not current_app.config.get("ENABLE_SESSION_FILES"):
|
||||
return jsonify({"ok": False, "error": "Not found"}), 404
|
||||
files = get_files(sid)
|
||||
if files is None:
|
||||
return jsonify({"ok": False, "error": "Session not found"}), 404
|
||||
@@ -118,6 +121,9 @@ def process_stream(sid):
|
||||
|
||||
all_files = [(fname, content, "") for fname, content in files]
|
||||
log.info("process_stream: start sid=%s files=%d", sid, len(all_files))
|
||||
# A failed rerun must not leave an older ZIP or mapping downloadable.
|
||||
store_result(sid, None)
|
||||
store_csv(sid, None)
|
||||
# Сессия живёт, пока идёт обработка (TTL возобновляется в finally генератора)
|
||||
pause_ttl(sid)
|
||||
|
||||
@@ -172,9 +178,16 @@ def process_stream(sid):
|
||||
log.info("worker: done sid=%s in %.1fs tokens=%d llm_sec=%.1f zip_len=%d",
|
||||
sid, dt, llm.tokens_total, llm.llm_sec, len(zip_data))
|
||||
q.put(("result", zip_data, csv_str, stats))
|
||||
except Exception as e:
|
||||
log.error("worker: exception sid=%s: %r\n%s", sid, e, traceback.format_exc())
|
||||
q.put(("error", repr(e)))
|
||||
except LLMIncompleteError as e:
|
||||
log.exception("worker: LLM incomplete sid=%s chunks=%d", sid, e.llm_errors)
|
||||
q.put(("error", {
|
||||
"error": "Processing stopped because the LLM could not process all fragments",
|
||||
"llm_incomplete": True,
|
||||
"llm_errors": e.llm_errors,
|
||||
}))
|
||||
except Exception:
|
||||
log.exception("worker: processing failed sid=%s", sid)
|
||||
q.put(("error", {"error": "Processing failed"}))
|
||||
|
||||
threading.Thread(target=worker, daemon=True).start()
|
||||
log.debug("process_stream: worker thread started sid=%s", sid)
|
||||
@@ -266,7 +279,7 @@ def process_stream(sid):
|
||||
store_csv(sid, csv_str)
|
||||
count = 0
|
||||
with zipfile.ZipFile(io.BytesIO(zip_data)) as zf:
|
||||
count = len([n for n in zf.namelist() if n != "mapping.csv"])
|
||||
count = len(zf.namelist())
|
||||
log.info("process_stream: complete sid=%s count=%d stats=%r", sid, count, stats)
|
||||
try:
|
||||
yield (
|
||||
@@ -295,10 +308,11 @@ def process_stream(sid):
|
||||
return
|
||||
|
||||
elif kind == "error":
|
||||
_, msg = evt
|
||||
log.error("process_stream: error event sid=%s msg=%r", sid, msg)
|
||||
_, error_data = evt
|
||||
log.error("process_stream: error event sid=%s llm_incomplete=%s",
|
||||
sid, error_data.get("llm_incomplete", False))
|
||||
try:
|
||||
yield f"event: proc_error\ndata: {json.dumps({'error': msg})}\n\n"
|
||||
yield f"event: proc_error\ndata: {json.dumps(error_data)}\n\n"
|
||||
except _disconnect_exceptions() as e:
|
||||
log.warning("process_stream: disconnect on error sid=%s err=%r", sid, e)
|
||||
return
|
||||
@@ -322,6 +336,8 @@ def process(sid):
|
||||
return jsonify({"ok": False, "error": "Session not found"}), 404
|
||||
if not files:
|
||||
return jsonify({"ok": False, "error": "No files"}), 400
|
||||
store_result(sid, None)
|
||||
store_csv(sid, None)
|
||||
try:
|
||||
llm = LLMClient()
|
||||
all_files = [(fname, content, "") for fname, content in files]
|
||||
@@ -334,9 +350,17 @@ def process(sid):
|
||||
if csv_str:
|
||||
store_csv(sid, csv_str)
|
||||
return jsonify({"ok": True, "status": "done"})
|
||||
except Exception as e:
|
||||
traceback.print_exc()
|
||||
return jsonify({"ok": False, "error": str(e)}), 500
|
||||
except LLMIncompleteError as e:
|
||||
log.exception("process: LLM incomplete sid=%s chunks=%d", sid, e.llm_errors)
|
||||
return jsonify({
|
||||
"ok": False,
|
||||
"error": "Processing stopped because the LLM could not process all fragments",
|
||||
"llm_incomplete": True,
|
||||
"llm_errors": e.llm_errors,
|
||||
}), 500
|
||||
except Exception:
|
||||
log.exception("process: processing failed sid=%s", sid)
|
||||
return jsonify({"ok": False, "error": "Processing failed"}), 500
|
||||
|
||||
|
||||
@api_bp.route("/download/<sid>", methods=["GET"])
|
||||
|
||||
@@ -457,7 +457,7 @@
|
||||
<div class="card-header">Обфускация документов</div>
|
||||
<div class="card-body">
|
||||
<p class="sub">
|
||||
Загрузите документы (.docx, .doc, .pdf, .txt, .md, .zip) — получите ZIP с обезличенными копиями.<br>
|
||||
Загрузите документы (.docx, .doc, .pdf, .txt, .md, .csv, .zip) — получите ZIP с обезличенными копиями.<br>
|
||||
CSV с таблицей замен скачивается отдельно.<br>
|
||||
<span style="color:#c0392b;">Ограничения: один файл не более 50 МБ, суммарно не более 500 МБ. Файлы сверх
|
||||
лимитов помечаются статусом «пропущен» и не участвуют в обфускации. Пустые, повреждённые файлы и сканы без
|
||||
@@ -507,7 +507,7 @@
|
||||
</div>
|
||||
<script src="/file-picker/file-picker.iife.js?v={{ version }}"></script>
|
||||
<script>
|
||||
const DOC_EXTS = ['.pdf', '.doc', '.docx', '.txt', '.md'];
|
||||
const DOC_EXTS = ['.pdf', '.doc', '.docx', '.txt', '.md', '.csv'];
|
||||
const ub = document.getElementById('uploadBtn');
|
||||
const cb = document.getElementById('cancelBtn');
|
||||
const nb = document.getElementById('newSessionBtn');
|
||||
@@ -853,8 +853,14 @@
|
||||
const d = JSON.parse(e.data);
|
||||
finishProcUI();
|
||||
setBusy(false);
|
||||
sessionDone = false;
|
||||
db.classList.remove('show');
|
||||
document.getElementById('statsBlock').classList.remove('show');
|
||||
nb.style.display = 'none';
|
||||
st.className = 'status error';
|
||||
st.textContent = 'Ошибка обработки: ' + (d.error || 'неизвестная ошибка');
|
||||
st.textContent = d.llm_incomplete
|
||||
? 'Обработка остановлена: LLM не обработал ' + d.llm_errors + ' фрагментов; результат не выдан'
|
||||
: 'Ошибка обработки: ' + (d.error || 'неизвестная ошибка');
|
||||
resolve();
|
||||
});
|
||||
activeES.onerror = function () {
|
||||
@@ -926,7 +932,7 @@
|
||||
onclick="event.stopPropagation()">
|
||||
<h3 style="margin-bottom:12px">Obfuscator — обфускация документов</h3>
|
||||
<p style="font-size:13px;color:var(--muted);line-height:1.6">
|
||||
Загрузите документы (.docx, .doc, .pdf, .txt, .md, .zip) — получите ZIP-архив с обезличенными копиями.
|
||||
Загрузите документы (.docx, .doc, .pdf, .txt, .md, .csv, .zip) — получите ZIP-архив с обезличенными копиями.
|
||||
Персональные данные, реквизиты компаний, телефоны, email и другие приватные сведения заменяются на обезличенные
|
||||
коды.
|
||||
CSV-файл с таблицей соответствия оригинал→замена скачивается отдельно.
|
||||
|
||||
@@ -5,8 +5,8 @@ from typing import Optional
|
||||
from .state import _sessions, _lock
|
||||
|
||||
|
||||
def store_csv(sid: str, csv_str: str) -> bool:
|
||||
"""Сохранить CSV с таблицей замен.
|
||||
def store_csv(sid: str, csv_str: Optional[str]) -> bool:
|
||||
"""Сохранить mapping CSV или инвалидировать его, передав None.
|
||||
|
||||
Returns:
|
||||
True если сохранено, False если сессии нет.
|
||||
|
||||
@@ -5,8 +5,8 @@ from typing import Optional
|
||||
from .state import _sessions, _lock
|
||||
|
||||
|
||||
def store_result(sid: str, zip_data: bytes) -> bool:
|
||||
"""Сохранить результат обработки (ZIP-архив).
|
||||
def store_result(sid: str, zip_data: Optional[bytes]) -> bool:
|
||||
"""Сохранить ZIP-результат или инвалидировать его, передав None.
|
||||
|
||||
Returns:
|
||||
True если сохранено, False если сессии нет.
|
||||
|
||||
@@ -77,7 +77,8 @@ def create_upload_refs_blueprint(cfg: dict = None) -> Blueprint:
|
||||
added = 0
|
||||
transport = cfg.get("httpxTransport") or current_app.config.get("UPLOAD_HTTPX_TRANSPORT")
|
||||
try:
|
||||
with httpx.Client(transport=transport, timeout=pull_timeout, follow_redirects=True) as client:
|
||||
# Redirects could bypass validation against the trusted VM-buffer prefix.
|
||||
with httpx.Client(transport=transport, timeout=pull_timeout, follow_redirects=False) as client:
|
||||
for ref in refs:
|
||||
name = safe_name(ref.get("name") or "")
|
||||
url = ref.get("url") or ""
|
||||
|
||||
Reference in New Issue
Block a user