How to Parse PDFs into RAG-Ready JSON with Unstructured
Unstructured is a document parser that splits PDFs, DOCX, HTML and scans into typed elements — Title, NarrativeText, Table, ListItem — each with page numbers and section metadata. You install the Python library, call partition, and send the JSON into a vector store instead of raw text.
That is the whole idea. Everything below is about making it behave on files that were not designed to be parsed.
Why raw PDF text breaks your retrieval pipeline
PDF is a print format. It stores glyphs at coordinates, not paragraphs. Extract the text naively and you get a blast of words with headers glued to body copy, tables flattened into lines of numbers, and footers repeated on every page.
Chunk that by character count and a 512-token window will happily cut through the middle of a fee schedule. Your retriever then returns half a table, and the model answers with half an answer.
Unstructured’s bet is that structure is recoverable before chunking. It labels each block, so chunking can respect section boundaries instead of guessing.
How do you install Unstructured?
pip install "unstructured[pdf]"
sudo apt-get install -y poppler-utils tesseract-ocr libmagic-dev # macOS: brew install
The extra matters. unstructured[pdf] pulls the PDF dependencies; plain pip install unstructured gives you a library that cannot open a PDF.
Install cost is the first real friction point. Detectron2, poppler and Tesseract push a local environment past 2 GB on some machines. Docker images built from this run large. If you only need to parse a few hundred pages a month, that is tolerable. If you are wiring it into CI, look at the hosted API instead.
What does the partition call actually return?
from unstructured.partition.auto import partition
elements = partition(
filename="10-k-2025.pdf",
strategy="hi_res",
include_page_breaks=True,
)
for el in elements[:3]:
print(el.category, "|", el.metadata.page_number, "|", str(el)[:80])
partition autodetects file type. Each element is a typed object: Title, NarrativeText, ListItem, Table, Header, Footer. str(el) gives the text, el.to_dict() gives text plus metadata including page number and bounding box coordinates.
The strategy argument is the one that decides quality and cost:
| Strategy | Speed | Good at | Weak at |
|---|---|---|---|
fast |
seconds | clean digital PDFs | anything scanned |
hi_res |
10-60s per doc | scans, multi-column, tables | heavy dependency footprint |
ocr_only |
medium | image-only files | layout recovery |
Start with fast. Switch to hi_res only for the files that fail, and check the output by eye before trusting it.
How do you chunk by section instead of by character count?
from unstructured.chunking.title import chunk_by_title
chunks = chunk_by_title(
elements,
max_characters=1200,
combine_text_under_n_chars=200,
new_after_n_chars=1000,
)
chunk_by_title cuts at section boundaries first, then applies the size cap. Metadata from the source element stays attached, so every chunk carries the section it came from. combine_text_under_n_chars merges orphaned fragments, which removes the one-line chunks that pollute a vector index.
This is the step most people skip. Naive character splitting is why retrieval feels random on contract documents.
What does Unstructured do badly?
Tables. Even hi_res loses structure on merged cells and multi-page tables, especially in financial filings where a single table spans three pages. The output keeps the text but the grid can come out flattened. Test your worst document before committing.
Scanned handwriting is another weak spot. Tesseract handles printed type; low-quality scans and cursive push you toward a paid vision model, and accuracy still disappoints on slide decks and image-heavy brochures.
Per-page pricing also creeps up. Hosted parsing is quoted around $0.003 per page for standard work and higher for high-resolution OCR. At 200,000 documents a month, that line item is a budget conversation, not a footnote.
Who should use it, and who should not
Use Unstructured if your source corpus is PDF-heavy and your retrieval quality is mediocre for structural reasons. Contracts, filings, policy manuals, scanned archives.
Skip it if your content is already HTML or Markdown, if you have clean JSON APIs behind the documents, or if you only have a few dozen well-formatted files. Parsing machinery for clean input is pure overhead.
For adjacent tooling, see the Unstructured tool page, and if you are assembling the rest of a retrieval stack, Crawl4AI covers the web-side equivalent of this job.
Quick Summary
- Unstructured turns PDFs, DOCX, HTML and scans into typed JSON elements with page numbers and coordinates.
- Install with
pip install "unstructured[pdf]"plus poppler and Tesseract, or the library cannot read PDFs. - Use
faststrategy first; reservehi_resfor scans and complex layouts. chunk_by_titlecuts at section boundaries and keeps section metadata on every chunk.- Tables and handwriting remain unreliable; hosted parsing runs near $0.003 per page.
