How to Extract Structured Web Data with the Diffbot Knowledge Graph API

Scraping the web reliably is hard because every site has a different layout, and layouts change without warning. Diffbot solves that by using computer vision and NLP to extract structured web data automatically. Instead of hand-writing parsers, you call an API and get clean JSON. This guide shows how to pull structured content and query the Diffbot Knowledge Graph for research and enrichment.

Understand the Diffbot APIs

Diffbot offers a few core products. The Analyze API reads any URL and classifies the page while extracting entities. Product, Article, and other specialized APIs return deeply structured fields for specific page types. The Knowledge Graph lets you query billions of linked entities. Deciding which you need first keeps your work focused and your costs predictable.

Step 1: Get Your Token and Try the Analyze API

Sign up at Diffbot and locate your developer token. Run a quick request to the Analyze endpoint with the token and a sample URL to see the returned JSON. You will notice the API classifies the page type and extracts key fields such as title, author, and text. This is the fastest way to confirm parsed output before you build anything.

Step 2: Extract Structured Data From a Single Page

Point the appropriate Diffbot endpoint at a real page, such as a product listing or an article. The response includes typed fields relevant to that content type. Store the returned JSON as-is rather than flattening it early, because you can reshape it later. If a field you need is missing, check whether the page uses dynamic JavaScript, which may change how extraction behaves.

Step 3: Set Up a Crawlbot Crawl for Many Pages

When you need data from many URLs, define a crawl in the Diffbot Crawlbot rather than looping over single requests. Give the crawl a seed URL or sitemap, a name, and a processing endpoint. Crawlbot follows links within your rules and sends each page through the extraction API automatically. Monitor the crawl queue and pause it whenever you want to adjust rules or check output quality.

Step 4: Query the Knowledge Graph

For entity-level research, use the Diffbot Knowledge Graph query API. You supply a search or a structured query describing an entity type and its properties, and Diffbot returns matching entities with their linked facts. This works well for market research, lead lists, and competitive intelligence because results are normalized rather than raw page text.

Step 5: Clean and Store Your Results

Pipeline the returned JSON into your database, deduplicating by a stable identifier such as a canonical URL or entity ID. Validate required fields and drop empty rows early. Because Diffbot returns consistent structure, loading into a relational store or a data warehouse is straightforward. Schedule incremental crawls so your dataset stays fresh.

Best Practices

Respect rate limits and crawl responsibly by reviewing Diffbot documentation for your plan. Test on a handful of representative pages before launching large crawls, and watch for pages that need JavaScript rendering. Keep your token secure. Finally, remember that extracted data reflects the source site, so run your own quality checks for fields that drive important decisions.