How to Keep Browse AI Robots Running When a Site Changes
Browse AI robots are easy to create and easy to forget. The ones that matter — the price sheet someone checks every morning, the competitor list in a weekly report — quietly stop working when a site ships a redesign, and nobody notices for three weeks because the run still returns “success”.
This is the maintenance side, which is where most of the real effort in scraping lives.
How do you know a robot is still working?
Success status is not the signal. Add three health checks to the output, in the sheet itself:
- Row count — expect a range per run (“180 to 240 rows”), alert outside it.
- Null rate per column — above a few percent on a column that is normally full means the field moved.
- Freshness — a
scraped_attimestamp per row; if the newest row is older than two schedule intervals, the robot stopped.
A single Google Sheets conditional alert on null rate catches more breakage than any monitoring dashboard, because it fires before you look.
How do you tell a redesign from a block?
Different fixes. Check the last run’s screenshot or HTML snapshot first:
| Symptom | Likely cause | Fix |
|---|---|---|
| All fields empty, page loads | Redesign | Re-train field selectors |
| Captcha or challenge page | Bot protection | Fewer runs, slower schedule |
| Login page | Session expired | Re-authenticate the robot |
| Partial columns | A field moved | Retrain that field only |
| Row count zero, no error | Wrong pagination control | Re-record the next-page click |
Re-training a few fields is a ten-minute job. Retraining the whole robot because you cannot tell the difference from a block wastes a morning.
How should you budget run quota?
Quota is consumed per page, not per run. A daily run over six paginated pages is six units a day, roughly 180 a month. Free and entry plans have limited monthly units; heavy monitoring goes to higher tiers.
Three rules that keep the budget sane:
- Scrape what changes. Detail pages for items whose price has not moved in a year are wasted units.
- Split schedules. Hourly for the 20 items that matter, daily for the rest.
- Pause instead of delete. A paused robot keeps its training; a deleted one costs a re-train later.
What is the realistic failure rate?
For a stable marketplace, a robot typically stays healthy for months. For a large e-commerce site that ships UI changes regularly, expect to touch it every three to four weeks.
Plan accordingly: if you own ten robots across five sites, budget an hour a week for maintenance. That is the honest number, and it is smaller than doing the same work by hand.
Where this does not work at all
Sites that rotate their layout per experiment — A/B-tested product pages change element structure between visits, and no fixed training survives that.
Anything you would need hourly with pagination at scale. Per-page quota plus browser-based execution makes that expensive fast; a scripted scraper is cheaper there.
Legal and ethical context matters too: robots.txt and terms of service still apply, and scraping personal data has rules that vary by jurisdiction.
Quick Summary
- Success status is not health — alert on row-count range, per-column null rate and a scraped_at timestamp.
- Distinguish redesign (retrain fields) from blocks (slow down) and expired sessions (re-authenticate) before rebuilding.
- Quota is consumed per page; split hourly for critical items and daily for the rest, and pause rather than delete.
- Budget about an hour a week of maintenance for ten robots across five sites.
- A/B-tested layouts and hourly scale crawling defeat fixed training; scripted scrapers fit those cases.
