refactor: vwts_scraper — proper Python package structure
- Split single file into 8 focused modules (399 lines total): config.py — constants fetch.py — HTTP GET with retries paginate.py — page_count + HEAD verification parse.py — HTML parsing (topics, posts, tags) state.py — per-section state.json output.py — JSONL with file rotation run.py — main orchestration loop __main__.py — CLI entry point - Run: python -m vwts_scraper <URL> - Page count method fully tested (13 sections, 100+ topics, 0 errors) - Resume support (Ctrl+C → state saved) - Per-section state files (no conflicts between sections)
This commit is contained in:
@@ -0,0 +1,13 @@
|
||||
"""Константы."""
|
||||
|
||||
import socket
|
||||
|
||||
# Глобальный таймаут сокетов
|
||||
socket.setdefaulttimeout(30)
|
||||
|
||||
DELAY = 1.5 # секунд между запросами
|
||||
TIMEOUT = (10, 30) # (connect, read)
|
||||
TOPICS_PER_FILE = 100
|
||||
HEADERS = {
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
|
||||
}
|
||||
Reference in New Issue
Block a user