How to Scrape a Docs Site with Firecrawl MCP into a RAG Index
Crawling a documentation site into a retrieval index with Firecrawl MCP takes four decisions: which paths to include, how deep to go, how to chunk the Markdown, and what metadata to keep. Get those right and answers cite real pages instead of blending three of them.
This is the pipeline I ran on a 340-page product docs site. Total cost was a few hundred credits and about 20 minutes of wall time.
Why crawl docs at all instead of just pasting a URL?
Because a single scrape of the homepage returns navigation and marketing copy, not the API reference. Docs sites hide the useful content behind a tree of pages that only a crawl can walk in order.
An indexed crawl also lets an agent answer version-specific questions. If you keep the version in the URL path, filters make that trivial.
Step 1: crawl with an include filter, not the whole domain
Bare-domain crawls pull in blog posts, changelogs and legal pages. Restrict to the docs path — typically /docs/ or /reference/ — and skip login, changelog and 404 pages.
Useful constraints for a starting run:
- Page limit: 200 to 400 to start, not 5000
- Paths to exclude:
/blog/,/changelog/,?utm,/print/ - Only main content: on, so you get the article without the sidebar
- Wait for JS: on for any docs site built with a client-side router
Run the crawl through the MCP tool from your agent, or POST the crawl endpoint directly if you want to script it. The job is asynchronous — poll until the status is completed, then download the results as JSONL.
Step 2: check the output before indexing anything
The failure mode nobody catches: half the pages are the same content under different URLs (/docs/auth, /docs/auth/, /docs/auth/index.html). Deduplicate by a normalised URL before chunking, or your retrieval will return the same passage three times.
Also eyeball one page with a code block. If indentation collapsed, the code samples are now garbage and your agent will write broken examples from them.
Step 3: chunk by heading, not by character count
Fixed 1000-token chunks cut API signatures in half. Chunk on level-2 and level-3 heading boundaries instead, with a target of 300 to 700 tokens per chunk and a short header prefix that names the page and section.
Store up to three metadata fields per chunk:
source_urlsection_titlepage_version
The section title matters more than people expect. A chunk that starts with “Rate limits” retrieves well; the same text without it competes with every other passage in the index.
Step 4: let Firecrawl MCP refresh the index on a schedule
Docs change monthly. Rather than a full re-crawl, run an incremental crawl weekly with the same include filter, hash each chunk, and only re-embed chunks whose hash changed. On the 340-page test site that was 12 to 30 changed chunks per week — cheap enough to run from a cron job.
Full re-crawls of that site cost roughly 340 credits. Weekly incremental runs cost under 40.
What does this pipeline not handle well?
Interactive examples. Pages with tabs (cURL / Python / Node) often flatten into one block, and the agent cannot tell which tab a snippet came from. If your docs use tabs heavily, capture each tab as a separate variant if the scraper exposes it; otherwise accept some noise.
API reference generated at runtime. Some reference pages build themselves from JavaScript. Without waiting for network idle you capture a loading spinner. Test three such pages before trusting the crawl.
Images and diagrams. Markdown gives you the image URL, not the content. Anything explained only in a diagram is invisible to retrieval.
Which setup fits which team
A single-product startup with 100 docs pages can run this as a one-off script and refresh manually each release. A documentation team with multiple versions should automate the version filter, or the agent will answer v3 questions with v4 syntax — the most common complaint I see in support channels using docs bots.
Skip the whole pipeline if your docs are under 20 pages. Pasting them into the agent context is simpler and more accurate.
Quick Summary
- Crawl the docs path only, with a 200-400 page limit and main-content extraction on.
- Deduplicate URLs before chunking; duplicate paths are the top indexing failure.
- Chunk by heading at 300-700 tokens and store source_url, section_title and version.
- Incremental weekly re-crawls cost under 40 credits versus ~340 for a full run.
- Tabbed code samples and client-rendered reference pages are the weak spots.
