This builds on the basic scrape workflow for Hyperbrowser. It is for developers wiring a model to fresh web content.

Why scrape instead of using a search API?

Search APIs return snippets — a sentence or two per result. That is often not enough for a factual answer, and the snippet may be stale. Scraping gives you the full page text, which is what the model needs to answer a specific question rather than summarize a headline.

Cost is the trade-off. A full browser scrape costs far more per page than a search query, so use search to pick the URLs and scraping only on the pages that make the cut.

How do you keep citations intact?

Attach the source URL to every chunk at fetch time, not after chunking. The pipeline:

1. Scrape each URL to markdown and keep the URL beside the text.
2. Split into chunks of roughly 500 tokens with a small overlap.
3. Tag every chunk with its source URL and page title.
4. Pass the chunks to the model with instructions to cite the tag it used.

Do the tagging before chunking and you never lose track of which passage came from where. Lose the mapping and the agent will cite a page it never read.

What breaks a research agent first?

Paywalls and redirects. A scraped page that turned out to be a login wall returns marketing copy, and the model answers from that. Detect it before the model sees it:

• Drop pages under a minimum word count — a 50-word result is almost always a wall.
• Flag pages whose title contains “sign in” or “subscribe”.
• Keep the final URL after redirects and discard off-topic domains.

Token bloat is the second failure. Ten long pages overflow the context window, so retrieve the relevant chunks per question rather than dumping every page into the prompt.

How many pages should one query scrape?

Five to ten is the practical range. Below five, coverage is thin and the agent misses the answer. Above ten, you pay for browser time and bury the useful chunk under noise. Pick the top results by relevance score and stop.

How do you make answers trustworthy?

Force a refusal path. Instruct the model to say it cannot answer when no chunk supports the claim, and show the cited URL next to every statement. An agent that always has an answer is an agent that fabricates.

Log the URL set per query too. When an answer looks wrong, the logs tell you whether the scrape missed the page or the model misread it — the same retrieval-versus-generation split that makes debugging possible.

Quick Summary

• Use search to pick URLs, Hyperbrowser to fetch full page text.
• Tag chunks with their source URL before splitting, not after.
• Filter paywalls and login walls by word count and title.
• Scrape 5–10 pages per query and retrieve only relevant chunks.
• Require refusal when no source supports an answer, and log the URL set.

It sits between a simple scraper and a full Playwright setup you maintain yourself. You give up control of the browser environment and skip the part where Chrome crashes on a server at 3 a.m.

How do you scrape your first page?

1. Create an account and copy your API key.
2. Call the scrape endpoint with a target URL and output format.
3. Read the returned markdown or HTML.

A single scrape request looks like this:

POST https://api.hyperbrowser.ai/api/scrape
Authorization: Bearer YOUR_KEY
{ "url": "https://example.com", "formats": ["markdown"] }

Markdown output is the reason to use it. You get readable text stripped of navigation and ads, which is exactly what a downstream LLM wants.

What kinds of sites does it handle?

Anything that needs a real browser:

• Single-page apps that render content in JavaScript.
• Infinite-scroll pages where you need more than the first screen.
• Pages behind a simple interaction, like clicking a tab to load a table.

If your target is static server-rendered HTML, a plain request library is cheaper and faster. Reaching for a browser API on a static page is paying for capacity you do not need.

Is Hyperbrowser free?

There is a free tier with a monthly allowance of credits or browser minutes, then usage-based pricing for higher volume. Browser time is the unit that matters — one minute of headless browsing costs far more than one HTTP request. A scrape of a heavy site can take several seconds, so small allowances disappear faster than you expect.

What will get you rate-limited?

Hammering one domain in parallel is the fastest route to blocks. Use their session and proxy features to rotate, and keep concurrency per domain low. Two requests at a time against the same host is usually safe; twenty will get the IP banned and waste your credits on challenge pages.

Also respect robots.txt and terms of service. Automated browser access is not a licence to ignore them, and a blocked account is harder to fix than a slower scraper.

Where does it get expensive?

Large crawls. Scraping ten pages is cheap; scraping ten thousand is a real bill, because each page spins a browser session. For anything more than a few hundred pages, evaluate whether a static-request path can cover most URLs and reserve the browser API for the pages that genuinely need JavaScript.

Quick Summary

• Hyperbrowser runs headless browsers in the cloud and returns markdown, HTML or data.
• Best for JavaScript-rendered pages and single-page apps.
• Free tier with a monthly allowance, then usage-based pricing by browser time.
• Rotate sessions and limit concurrency to avoid IP blocks.
• Large crawls get costly — use plain requests for static pages.