Nail/LLM-UI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
10 Commits 2 Branches 0 Tags
5c142096de9eb1ce9282ca08e8778672e05eee5c
Commit Graph
10 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
“Naeel” 5c142096de scraper: новая архитектура — core/ + forums/ (BaseAdapter, XenForo, phpBB) 
- core/ — общий слой: fetch, throttle, output, state, runner
- forums/base.py — абстрактный BaseAdapter
- forums/xenforo/ — адаптер для XenForo (vwts.ru)
- forums/phpbb/ — адаптер для phpBB (нива-лада.рф)
- __main__.py — точка входа CLI
- history/001-initial-structure.md — журнал изменений

Также на ВМ (не в этом коммите):
- lada_scraper/parse.py — пропуск sticky-тем при многопоточном парсинге
 2026-06-06 11:46:57 +04:00
“Naeel” 673bfe5e2a .gitignore: фикс vwts_scraper.py + lada_data 2026-06-04 11:47:56 +03:00
“Naeel” eca5e3cda6 lada_scraper: парсер phpBB форума нива-лада.рф + доку 2026-06-04 11:47:02 +03:00
“Naeel” 928c979509 docs: mercedes + обновление forums-survey 2026-06-04 10:50:58 +03:00
“Naeel” c3211f0602 feat: многопоточный парсинг + forum survey 
- Throttle: Thread-safe координация задержек
- BanDetector: автоопределение бана
- fetch: thread-local сессии вместо глобальной
- run: ThreadPoolExecutor + диапазоны страниц + мерж
- output/state: поддержка worker-папок и merge_states()
- __main__: --workers, --no-delay, --no-ban-test
- VWTS/forums-survey.md: обзор 20+ автофорумов
 2026-06-04 10:48:57 +03:00
“Naeel” 22ca83adf8 feat: new UI with sidebar, favicon, copilot instructions, gitignore cleanup 2026-06-04 09:10:33 +03:00
“Naeel” 4a3d5d21d1 fix: review findings - OUTPUT_DIR centralised, no hardcoded vwts.ru, normal imports 2026-06-04 09:08:51 +03:00
“Naeel” 1cfa6d1595 fix: audit findings — edge cases in paginate, output, run 
- paginate.py: max(nums) on empty set → guard
- paginate.py: HEAD fallback to GET if server doesn't support HEAD
- output.py: disk health check every 500 topics
- run.py: avoid page-1 extra redirect
- __main__.py: URL validation (must contain vwts.ru)
 2026-06-04 08:58:09 +03:00
“Naeel” 010b10c43e refactor: vwts_scraper — proper Python package structure 
- Split single file into 8 focused modules (399 lines total):
  config.py — constants
  fetch.py — HTTP GET with retries
  paginate.py — page_count + HEAD verification
  parse.py — HTML parsing (topics, posts, tags)
  state.py — per-section state.json
  output.py — JSONL with file rotation
  run.py — main orchestration loop
  __main__.py — CLI entry point
- Run: python -m vwts_scraper <URL>
- Page count method fully tested (13 sections, 100+ topics, 0 errors)
- Resume support (Ctrl+C → state saved)
- Per-section state files (no conflicts between sections)
 2026-06-04 08:55:23 +03:00
“Naeel” 9f547d8d9e first commit 2026-06-03 15:55:08 +03:00