What it does
Crawls Deepwiki documentation URLs, extracts all linked pages, and converts them to Markdown. Supports two output modes: a single merged document or structured page-by-page data. The crawler sanitizes HTML (removing headers, footers, navigation, scripts, and ads) and rewrites links for Markdown compatibility. Domain filtering restricts input to deepwiki.com only.
Who it's for
Developers and AI tool builders who need to extract and convert Deepwiki documentation into structured Markdown for embedding in knowledge bases, search indexes, or language model contexts. Useful for anyone automating documentation processing workflows.
Common use cases
- Convert a complete Deepwiki repository to a single Markdown file for offline use or archival
- Fetch structured page data to build searchable documentation indexes for LLM systems
- Extract framework documentation (shadcn-ui, Tailwind, etc.) as Markdown for knowledge bases
Setup pitfalls
- Server is currently non-functional. DeepWiki has disabled scraping; the README recommends using the official Deepwiki MCP server at
docs.devin.aiinstead. - High risk classification—reads and writes the filesystem and makes network calls, requiring careful sandboxing in untrusted environments.
- CI tests are failing and the codebase is 84 days stale, indicating maintenance gaps.
- Configure
DEEPWIKI_MAX_CONCURRENCY,DEEPWIKI_REQUEST_TIMEOUT, and retry settings via environment variables to tune behavior.