How to Build an AI Research Agent on Hyperbrowser Data
An AI research agent on Hyperbrowser works by scraping a set of live pages to markdown, chunking that text with source URLs attached, and asking an LLM to answer only from those chunks with citations. The scraping layer is what keeps the model grounded in current pages instead of training data.
This builds on the basic scrape workflow for Hyperbrowser. It is for developers wiring a model to fresh web content.
Why scrape instead of using a search API?
Search APIs return snippets — a sentence or two per result. That is often not enough for a factual answer, and the snippet may be stale. Scraping gives you the full page text, which is what the model needs to answer a specific question rather than summarize a headline.
Cost is the trade-off. A full browser scrape costs far more per page than a search query, so use search to pick the URLs and scraping only on the pages that make the cut.
How do you keep citations intact?
Attach the source URL to every chunk at fetch time, not after chunking. The pipeline:
- Scrape each URL to markdown and keep the URL beside the text.
- Split into chunks of roughly 500 tokens with a small overlap.
- Tag every chunk with its source URL and page title.
- Pass the chunks to the model with instructions to cite the tag it used.
Do the tagging before chunking and you never lose track of which passage came from where. Lose the mapping and the agent will cite a page it never read.
What breaks a research agent first?
Paywalls and redirects. A scraped page that turned out to be a login wall returns marketing copy, and the model answers from that. Detect it before the model sees it:
- Drop pages under a minimum word count — a 50-word result is almost always a wall.
- Flag pages whose title contains “sign in” or “subscribe”.
- Keep the final URL after redirects and discard off-topic domains.
Token bloat is the second failure. Ten long pages overflow the context window, so retrieve the relevant chunks per question rather than dumping every page into the prompt.
How many pages should one query scrape?
Five to ten is the practical range. Below five, coverage is thin and the agent misses the answer. Above ten, you pay for browser time and bury the useful chunk under noise. Pick the top results by relevance score and stop.
| Pages | Effect |
|---|---|
| 1–3 | Fast and cheap, often incomplete |
| 5–10 | Best balance for factual queries |
| 20+ | Costly, diminishing relevance, context overflow |
How do you make answers trustworthy?
Force a refusal path. Instruct the model to say it cannot answer when no chunk supports the claim, and show the cited URL next to every statement. An agent that always has an answer is an agent that fabricates.
Log the URL set per query too. When an answer looks wrong, the logs tell you whether the scrape missed the page or the model misread it — the same retrieval-versus-generation split that makes debugging possible.
Quick Summary
- Use search to pick URLs, Hyperbrowser to fetch full page text.
- Tag chunks with their source URL before splitting, not after.
- Filter paywalls and login walls by word count and title.
- Scrape 5–10 pages per query and retrieve only relevant chunks.
- Require refusal when no source supports an answer, and log the URL set.
