naeel 010b10c43e refactor: vwts_scraper — proper Python package structure ...

- Split single file into 8 focused modules (399 lines total):
  config.py — constants
  fetch.py — HTTP GET with retries
  paginate.py — page_count + HEAD verification
  parse.py — HTML parsing (topics, posts, tags)
  state.py — per-section state.json
  output.py — JSONL with file rotation
  run.py — main orchestration loop
  __main__.py — CLI entry point
- Run: python -m vwts_scraper <URL>
- Page count method fully tested (13 sections, 100+ topics, 0 errors)
- Resume support (Ctrl+C → state saved)
- Per-section state files (no conflicts between sections)

2026-06-04 08:55:23 +03:00

2.4 KiB

Raw History

View Raw