I posted an early version of this here about a year ago. It crawls a documentation site and spits out clean Markdown plus a JSONL file, mainly for feeding docs into LLM pipelines.
A year of changes: it does incremental crawls now (only flags pages whose actual content changed, not the nav), keeps crawl history in SQLite, and can run as an MCP server so Claude Code or Cursor can read a crawled site locally instead of fetching it live every time. Still pure Go, no CGO (modernc.org/sqlite).
Repo: https://github.com/Sriram-PR/doc-scraper
The two things people asked for in the last thread are both in: per-page metadata, and single-file output (llms.txt / llms-full.txt).
Honestly most of the year went into bugs I only found because other people ran it. A Windows user hit actual data loss because finalization renamed over an open file handle, which Linux allows and Windows doesn't. And the sitemap code path turned out to be quietly ignoring the per-host rate limit. CI runs on all three OSes now.
Known gaps: static HTML only, no auth support yet.
If you try it on a site and the output comes out mangled, open an issue with the URL. Those reports have been the most useful thing for improving it.
[link] [留言]