naeel
010b10c43e
- Split single file into 8 focused modules (399 lines total): config.py — constants fetch.py — HTTP GET with retries paginate.py — page_count + HEAD verification parse.py — HTML parsing (topics, posts, tags) state.py — per-section state.json output.py — JSONL with file rotation run.py — main orchestration loop __main__.py — CLI entry point - Run: python -m vwts_scraper <URL> - Page count method fully tested (13 sections, 100+ topics, 0 errors) - Resume support (Ctrl+C → state saved) - Per-section state files (no conflicts between sections)
7 lines
65 B
Plaintext
7 lines
65 B
Plaintext
doc/
|
|
token*.*
|
|
|
|
# vwts.ru scraper
|
|
VWTS/
|
|
vwts_data/
|
|
vwts_scraper.py |