How to Scrape a Site With AgentQL Queries Instead of Selectors

November 13, 2026

AgentQL replaces brittle CSS selectors with a query written in plain language plus field names, and returns structured JSON. You run it through the REST API or inside Playwright, and a first working extraction typically takes 10-15 minutes. Free accounts get a monthly query allowance; beyond that pricing is per query.

AgentQL is aimed at the part of scraping that breaks: when the site adds a wrapper div and every selector you wrote returns null. A query describing the field survives most markup changes, which is the whole argument.

What does an AgentQL query look like?

A query is a block of field names with optional descriptions in brackets. It looks like a GraphQL selection set with English inside the braces.

{
  products[] {
    name
    price
    rating
    link
  }
}

The word before the brackets is the output key. Text in brackets describes the element in your own words, which is how you disambiguate two similarly named fields:

{
  products[] {
    name [the product title, not the brand]
    price [current selling price, including currency symbol]
  }
}

That bracket hint is the single highest-leverage part of the syntax. Without it, “name” on a listing page often grabs the site name from the header.

How do you run it in Playwright?

Install the Python package, wrap an existing Playwright page, and call the query method.

  1. pip install agentql playwright, then playwright install
  2. Get an API key from the AgentQL dashboard and set AGENTQL_API_KEY
  3. Launch a page with agentql.wrap(page)
  4. Page goto the target URL and wait for content
  5. Call page.query_elements(query) or page.query_data(query)
  6. Read the returned dict, or loop over the element handles

query_elements returns locators you can interact with; query_data returns plain data. Use query_data for extraction to JSON, and query_elements when you need to click a result afterwards.

Which sites does it fail on?

The ones with no semantic structure. A canvas-rendered app, an image-only product grid, or a page that hides everything behind a login will return empty results regardless of query wording.

Infinite scroll is the common practical failure. The query extracts what is currently in the DOM, so you need to scroll programmatically and re-query, or the result stops at the first 20 items. Add a scroll loop that runs until the row count stops growing.

Also expect trouble with pages where the same visual element serves four different purposes depending on state — a price field showing “Sold out” is not a price, and the model may still return the string.

How much does it cost?

Free tier gives a limited number of queries per month for evaluation. Paid plans are per-query, with volume tiers, and the exact rate is best checked in the dashboard because it has moved since launch.

Budget one query per page, plus retries. A 5,000-page scrape with a 20% retry rate is 6,000 queries. That multiplication is where people get surprised; a per-query price that looks trivial becomes a real line item at scale.

For high-volume repetitive scraping of a stable site, plain CSS selectors cost nothing and run faster. AgentQL’s economics work best at moderate volume on sites that change.

Does it replace selectors entirely?

No. A hybrid setup is what experienced teams actually run.

Situation Better choice
Stable site, high volume CSS selectors
Site changes layout often AgentQL
Deep pagination clicks Playwright selectors
Unstructured text extraction AgentQL

The hybrid pattern: use selectors to navigate and paginate, and a query to extract fields. You keep deterministic behaviour where it matters and resilience where it broke before.

What should you check before trusting the output?

Field completeness and type consistency. A price extracted as “$12.99” in one row and “12.99” in the next will break your downstream aggregation.

Three quick checks after a run:

  • Count rows against what you see on the page
  • Spot-check five rows field by field
  • Look for nulls clustered in one field — that signals a selector-scope problem, not bad data

Nulls in a single field are almost always a strong signal that your query hint needs to be more specific.

Who is AgentQL for?

Developers and data engineers maintaining scrapers that break every few weeks, and teams extracting from a handful of sites at moderate volume. Anyone building an agent that needs to read a web page without hand-crafted selectors.

Not for people who do not write code — a browser extension or a spreadsheet importer is the right tool there. Not for massive high-volume scraping where the per-query price dominates the budget.

Quick Summary

  • AgentQL extracts data using natural-language field queries instead of CSS selectors.
  • Add bracket hints to each field to prevent wrong-element grabs.
  • Runs via REST API or inside Playwright with agentql.wrap(page).
  • Infinite scroll and canvas-rendered pages need extra handling.
  • Per-query pricing favours moderate volume; selectors remain cheaper for stable sites.
Tags: