naeel
c3211f0602
feat: многопоточный парсинг + forum survey
...
- Throttle: Thread-safe координация задержек
- BanDetector: автоопределение бана
- fetch: thread-local сессии вместо глобальной
- run: ThreadPoolExecutor + диапазоны страниц + мерж
- output/state: поддержка worker-папок и merge_states()
- __main__: --workers, --no-delay, --no-ban-test
- VWTS/forums-survey.md: обзор 20+ автофорумов
2026-06-04 10:48:57 +03:00
naeel
22ca83adf8
feat: new UI with sidebar, favicon, copilot instructions, gitignore cleanup
2026-06-04 09:10:33 +03:00
naeel
4a3d5d21d1
fix: review findings - OUTPUT_DIR centralised, no hardcoded vwts.ru, normal imports
2026-06-04 09:08:51 +03:00
naeel
1cfa6d1595
fix: audit findings — edge cases in paginate, output, run
...
- paginate.py: max(nums) on empty set → guard
- paginate.py: HEAD fallback to GET if server doesn't support HEAD
- output.py: disk health check every 500 topics
- run.py: avoid page-1 extra redirect
- __main__.py: URL validation (must contain vwts.ru)
2026-06-04 08:58:09 +03:00
naeel
010b10c43e
refactor: vwts_scraper — proper Python package structure
...
- Split single file into 8 focused modules (399 lines total):
config.py — constants
fetch.py — HTTP GET with retries
paginate.py — page_count + HEAD verification
parse.py — HTML parsing (topics, posts, tags)
state.py — per-section state.json
output.py — JSONL with file rotation
run.py — main orchestration loop
__main__.py — CLI entry point
- Run: python -m vwts_scraper <URL>
- Page count method fully tested (13 sections, 100+ topics, 0 errors)
- Resume support (Ctrl+C → state saved)
- Per-section state files (no conflicts between sections)
2026-06-04 08:55:23 +03:00
naeel