<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Firecrawl MCP &#8211; iAIFeed</title>
	<atom:link href="https://www.iaifeed.com/tag/firecrawl-mcp/feed" rel="self" type="application/rss+xml" />
	<link>https://www.iaifeed.com</link>
	<description>iAIFeed</description>
	<lastBuildDate>Thu, 05 Nov 2026 13:00:48 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.5</generator>

<image>
	<url>https://www.iaifeed.com/wp-content/uploads/2026/07/cropped-iaifeed1-80x80.png</url>
	<title>Firecrawl MCP &#8211; iAIFeed</title>
	<link>https://www.iaifeed.com</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>How to Scrape a Docs Site with Firecrawl MCP into a RAG Index</title>
		<link>https://www.iaifeed.com/how-to-scrape-a-docs-site-with-firecrawl-mcp-into-rag</link>
					<comments>https://www.iaifeed.com/how-to-scrape-a-docs-site-with-firecrawl-mcp-into-rag#respond</comments>
		
		<dc:creator><![CDATA[iamltlb]]></dc:creator>
		<pubDate>Thu, 05 Nov 2026 12:47:30 +0000</pubDate>
				<category><![CDATA[Blog]]></category>
		<category><![CDATA[Firecrawl MCP]]></category>
		<guid isPermaLink="false">https://www.iaifeed.com/how-to-scrape-a-docs-site-with-firecrawl-mcp-into-rag</guid>

					<description><![CDATA[Crawl a documentation site with Firecrawl MCP, chunk the Markdown cleanly and load it into a local vector store for an agent that answers from your docs.]]></description>
										<content:encoded><![CDATA[<p>Crawling a documentation site into a retrieval index with Firecrawl MCP takes four decisions: which paths to include, how deep to go, how to chunk the Markdown, and what metadata to keep. Get those right and answers cite real pages instead of blending three of them.</p>
<p>This is the pipeline I ran on a 340-page product docs site. Total cost was a few hundred credits and about 20 minutes of wall time.</p>
<h2 id="why-crawl-docs-at-all-instead-of-just-pasting-a-url">Why crawl docs at all instead of just pasting a URL?</h2>
<p>Because a single scrape of the homepage returns navigation and marketing copy, not the API reference. Docs sites hide the useful content behind a tree of pages that only a crawl can walk in order.</p>
<p>An indexed crawl also lets an agent answer version-specific questions. If you keep the version in the URL path, filters make that trivial.</p>
<h2 id="step-1-crawl-with-an-include-filter-not-the-whole-domain">Step 1: crawl with an include filter, not the whole domain</h2>
<p>Bare-domain crawls pull in blog posts, changelogs and legal pages. Restrict to the docs path — typically <code>/docs/</code> or <code>/reference/</code> — and skip login, changelog and 404 pages.</p>
<p>Useful constraints for a starting run:</p>
<ul>
<li>Page limit: 200 to 400 to start, not 5000</li>
<li>Paths to exclude: <code>/blog/</code>, <code>/changelog/</code>, <code>?utm</code>, <code>/print/</code></li>
<li>Only main content: on, so you get the article without the sidebar</li>
<li>Wait for JS: on for any docs site built with a client-side router</li>
</ul>
<p>Run the crawl through the MCP tool from your agent, or <code>POST</code> the crawl endpoint directly if you want to script it. The job is asynchronous — poll until the status is completed, then download the results as JSONL.</p>
<h2 id="step-2-check-the-output-before-indexing-anything">Step 2: check the output before indexing anything</h2>
<p>The failure mode nobody catches: half the pages are the same content under different URLs (<code>/docs/auth</code>, <code>/docs/auth/</code>, <code>/docs/auth/index.html</code>). Deduplicate by a normalised URL before chunking, or your retrieval will return the same passage three times.</p>
<p>Also eyeball one page with a code block. If indentation collapsed, the code samples are now garbage and your agent will write broken examples from them.</p>
<h2 id="step-3-chunk-by-heading-not-by-character-count">Step 3: chunk by heading, not by character count</h2>
<p>Fixed 1000-token chunks cut API signatures in half. Chunk on level-2 and level-3 heading boundaries instead, with a target of 300 to 700 tokens per chunk and a short header prefix that names the page and section.</p>
<p>Store up to three metadata fields per chunk:</p>
<ul>
<li><code>source_url</code></li>
<li><code>section_title</code></li>
<li><code>page_version</code></li>
</ul>
<p>The section title matters more than people expect. A chunk that starts with &#8220;Rate limits&#8221; retrieves well; the same text without it competes with every other passage in the index.</p>
<h2 id="step-4-let-firecrawl-mcp-refresh-the-index-on-a-schedule">Step 4: let <a href="https://www.iaifeed.com/ai-tool/firecrawl-mcp">Firecrawl MCP</a> refresh the index on a schedule</h2>
<p>Docs change monthly. Rather than a full re-crawl, run an incremental crawl weekly with the same include filter, hash each chunk, and only re-embed chunks whose hash changed. On the 340-page test site that was 12 to 30 changed chunks per week — cheap enough to run from a cron job.</p>
<p>Full re-crawls of that site cost roughly 340 credits. Weekly incremental runs cost under 40.</p>
<h2 id="what-does-this-pipeline-not-handle-well">What does this pipeline not handle well?</h2>
<p><strong>Interactive examples.</strong> Pages with tabs (cURL / Python / Node) often flatten into one block, and the agent cannot tell which tab a snippet came from. If your docs use tabs heavily, capture each tab as a separate variant if the scraper exposes it; otherwise accept some noise.</p>
<p><strong>API reference generated at runtime.</strong> Some reference pages build themselves from JavaScript. Without waiting for network idle you capture a loading spinner. Test three such pages before trusting the crawl.</p>
<p><strong>Images and diagrams.</strong> Markdown gives you the image URL, not the content. Anything explained only in a diagram is invisible to retrieval.</p>
<h2 id="which-setup-fits-which-team">Which setup fits which team</h2>
<p>A single-product startup with 100 docs pages can run this as a one-off script and refresh manually each release. A documentation team with multiple versions should automate the version filter, or the agent will answer v3 questions with v4 syntax — the most common complaint I see in support channels using docs bots.</p>
<p>Skip the whole pipeline if your docs are under 20 pages. Pasting them into the agent context is simpler and more accurate.</p>
<h2 id="quick-summary">Quick Summary</h2>
<ul>
<li>Crawl the docs path only, with a 200-400 page limit and main-content extraction on.</li>
<li>Deduplicate URLs before chunking; duplicate paths are the top indexing failure.</li>
<li>Chunk by heading at 300-700 tokens and store source_url, section_title and version.</li>
<li>Incremental weekly re-crawls cost under 40 credits versus ~340 for a full run.</li>
<li>Tabbed code samples and client-rendered reference pages are the weak spots.</li>
</ul>
]]></content:encoded>
					
					<wfw:commentRss>https://www.iaifeed.com/how-to-scrape-a-docs-site-with-firecrawl-mcp-into-rag/feed</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>How to Set Up Firecrawl MCP in Claude Code and Cursor</title>
		<link>https://www.iaifeed.com/how-to-set-up-firecrawl-mcp-in-claude-and-cursor</link>
					<comments>https://www.iaifeed.com/how-to-set-up-firecrawl-mcp-in-claude-and-cursor#respond</comments>
		
		<dc:creator><![CDATA[iamltlb]]></dc:creator>
		<pubDate>Thu, 05 Nov 2026 12:47:04 +0000</pubDate>
				<category><![CDATA[Blog]]></category>
		<category><![CDATA[Firecrawl MCP]]></category>
		<guid isPermaLink="false">https://www.iaifeed.com/how-to-set-up-firecrawl-mcp-in-claude-and-cursor</guid>

					<description><![CDATA[Firecrawl MCP gives your coding agent scrape, crawl and search tools over the Model Context Protocol. Config takes ten minutes and needs an API key from firecrawl.dev.]]></description>
										<content:encoded><![CDATA[<p>Firecrawl MCP is the official Model Context Protocol server for Firecrawl, the web scraping service. It exposes four main tools — scrape a URL, crawl a site, search the web, and extract structured data — so an agent can pull live web content without you writing fetch code.</p>
<p>Setup is one API key and one JSON block. The annoying part is choosing tools carefully, because an over-broad tool set makes agents slow and expensive.</p>
<h2 id="what-does-the-firecrawl-mcp-server-actually-expose">What does the Firecrawl MCP server actually expose?</h2>
<p>Four tools that cover most research needs:</p>
<ul>
<li><code>firecrawl_scrape</code> — fetch one URL and return Markdown, HTML or plain text</li>
<li><code>firecrawl_crawl</code> — start a job that walks a site up to a page limit and returns each page as Markdown</li>
<li><code>firecrawl_search</code> — web search that can optionally scrape the top results</li>
<li><code>firecrawl_extract</code> — pull structured fields from pages using a prompt and a JSON schema</li>
</ul>
<p>Responses are Markdown by default, which is the format LLMs handle best. That default alone saves you an HTML-stripping step in most pipelines.</p>
<h2 id="how-do-you-get-a-firecrawl-api-key">How do you get a Firecrawl API key?</h2>
<p>Create an account at firecrawl.dev, open the dashboard, and copy the key with the <code>fc-</code> prefix. The free tier gives you a few hundred credits to test; paid plans start around $16/month for Small. One page scrape generally costs 1 credit, and crawling a 50-page site costs roughly 50.</p>
<p>Keep the key out of your shell history. Put it in an environment variable and reference that variable from the MCP config, not the literal string.</p>
<h2 id="how-do-you-add-it-to-claude-code">How do you add it to Claude Code?</h2>
<p>The CLI path is the simplest:</p>
<pre><code class="language-bash">claude mcp add firecrawl -e FIRECRAWL_API_KEY=fc-your-key -- npx -y firecrawl-mcp
</code></pre>
<p>Then start Claude Code and run <code>/mcp</code> to confirm the server shows as connected and lists four tools. If it shows nothing, the usual cause is a broken <code>npx</code> cache — run <code>npx -y firecrawl-mcp</code> manually once to populate it.</p>
<p>For project-scoped setup, add the same entry under <code>mcpServers</code> in <code>.mcp.json</code> at the repo root. Use that when a teammate should get the server automatically.</p>
<h2 id="how-do-you-add-it-to-cursor-and-other-clients">How do you add it to Cursor and other clients?</h2>
<p>In <a href="https://www.iaifeed.com/ai-tool/firecrawl-mcp">Firecrawl MCP</a> the pattern is the same for every JSON-configured client. Open the MCP settings file (<code>~/.cursor/mcp.json</code> for Cursor) and add:</p>
<pre><code class="language-json">{
  &quot;mcpServers&quot;: {
    &quot;firecrawl&quot;: {
      &quot;command&quot;: &quot;npx&quot;,
      &quot;args&quot;: [&quot;-y&quot;, &quot;firecrawl-mcp&quot;],
      &quot;env&quot;: { &quot;FIRECRAWL_API_KEY&quot;: &quot;fc-your-key&quot; }
    }
  }
}
</code></pre>
<p>Restart the client. Cursor shows a green dot next to the server name when it started. VS Code with Copilot uses the same shape in <code>.vscode/mcp.json</code>.</p>
<h2 id="which-tools-should-you-enable-by-hand">Which tools should you enable by hand?</h2>
<p>All four, and agents get chatty. In my test, an agent asked to &#8220;compare three competitors&#8217; pricing&#8221; opened with a crawl of each homepage instead of a search, which burned about 400 credits on pages nobody read.</p>
<p>Two mitigations:</p>
<ol>
<li>Disable <code>firecrawl_crawl</code> for research-heavy projects and keep only scrape, search and extract.</li>
<li>In your agent instructions, add: &#8220;Use search before crawl. Never crawl more than 10 pages without asking.&#8221;</li>
</ol>
<p>Credits are the real constraint. An agent with a crawl tool and no instruction will exhaust a free tier in one session.</p>
<h2 id="what-breaks-most-often">What breaks most often?</h2>
<ul>
<li><strong>Node version.</strong> The server needs Node 18 or newer. Node 16 produces a silent start failure that looks like a config error.</li>
<li><strong>Rate limits.</strong> Free-tier accounts hit per-minute limits during crawls, and the agent reports the error as &#8220;empty content&#8221;.</li>
<li><strong>JavaScript-heavy sites.</strong> Default scraping misses content rendered on the client. Pass the wait-for parameter so the page has time to settle.</li>
<li><strong>Auth walls.</strong> Firecrawl cannot log in for you. Paywalled content returns a login page, not the article.</li>
</ul>
<h2 id="who-is-this-for">Who is this for</h2>
<p>Good fit: agents that need current web data — competitor research, docs checking, pricing comparison. Poor fit: anything where the source must be stable over time, since scraped pages change. If you need repeatable input, cache the Markdown you scraped instead of re-running the tool.</p>
<h2 id="quick-summary">Quick Summary</h2>
<ul>
<li>Firecrawl MCP is the official MCP server exposing scrape, crawl, search and extract tools.</li>
<li>Get an <code>fc-</code> API key at firecrawl.dev; free tier gives a few hundred test credits, paid from ~$16/month.</li>
<li>Install with <code>claude mcp add firecrawl -e FIRECRAWL_API_KEY=... -- npx -y firecrawl-mcp</code>, or a <code>mcpServers</code> block in Cursor.</li>
<li>Disable crawl for research work or agents burn 400+ credits per session.</li>
<li>Needs Node 18+; paywalled and JS-heavy pages are the common failure cases.</li>
</ul>
]]></content:encoded>
					
					<wfw:commentRss>https://www.iaifeed.com/how-to-set-up-firecrawl-mcp-in-claude-and-cursor/feed</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
	</channel>
</rss>
