How to Batch Validate AirOps Output Before It Publishes
You validate batch output from AirOps by adding a second workflow that scores each run on fixed rules before anything reaches your CMS. The eval runs on the same input plus the generated text, returns pass/fail plus a reason, and routes failures to a review sheet.
Most AirOps content failures are not creative. They are structural: wrong word count, missing keyword, a hallucinated statistic, or two paragraphs that say the same thing. Those are cheap to catch automatically and expensive to catch by reading.
Why does AirOps output drift on batch runs?
Because a workflow that scored 9/10 on your test prompt gets fed fifty real inputs, and a third of them are edge cases. A company name with an ampersand. A keyword nobody searches. A source URL that 404s mid-research.
AirOps agents chain LLM calls and web fetches. Once a fetch fails, the downstream step still runs, just with thinner context. The article reads fine. It is also slightly wrong, and you only find out after publishing.
The fix is not a better prompt. It is a gate between the workflow and your CMS.
Step 1: Define the pass criteria as data, not vibes
Write the rules down before you build the checker. A workable starting set for blog content:
- word count between 850 and 1200
- primary keyword in the H2 headings, at least once
- at least three sentences containing a number or a date
- no sentence over 45 words
- zero banned phrases (your brand list, plus generic filler like “in today’s world”)
- at least one internal link
Six to ten rules is the sweet spot. Below six you miss real failures; above fifteen you reject good drafts and your reviewers start ignoring the flags.
Step 2: Build the eval workflow in AirOps
Create a second workflow with two inputs: source_input (the original JSON you sent) and draft (the generated article). Add a single LLM step with structured output — AirOps supports JSON schema on agent output, which matters here because free-text feedback cannot be counted.
Ask for this shape:
Output shape: pass (boolean), score (0-100 integer), failures (array of rule names), notes (one short sentence)., “score”: 0-100, “failures”: [“rule name”], “notes”: “one short sentence” }
Give the model the rule list verbatim in the prompt and tell it to return the rule names exactly as written. Anything else and your aggregation later becomes string matching on synonyms.
Keep this step cheap. A short-context check on the draft alone costs a fraction of the generation workflow.
Step 3: Fan out over the batch, not one at a time
The AirOps bulk runner takes a CSV or a sheet and runs the workflow once per row. Point it at your eval workflow with one row per draft.
For 100 drafts expect 3 to 6 minutes of wall clock. Concurrency is managed by AirOps; you do not tune it, and pushing more rows than your credit balance allows just fails the overflow rows silently at the end.
Two operational notes from running this on real batches:
- Run the eval before the generation batch is archived, or you lose the link between source input and draft and cannot reproduce failures.
- Store the eval JSON next to the draft, not in a separate sheet. Joining two sheets by company name breaks the moment two clients share a name.
Step 4: Route by score, not by the pass flag
The boolean pass/fail is too blunt. Sort into three buckets instead:
| Score | Action |
|---|---|
| 90-100 | auto-queue to CMS as draft |
| 60-89 | send to editor queue with notes |
| 0-59 | regenerate with the failures appended to the prompt |
That top bucket is where the time savings live. On a real batch of 80 articles, typically 50 to 60 land above 90 once the generation prompt is tuned, and the rest need a human for five minutes each.
For the regenerate bucket, append the failure list to the original prompt and re-run once. A second attempt recovers roughly half of them. Do not loop three times — the cost climbs and the quality flattens.
What does this cost compared to reading everything?
A reviewer at 8 minutes per article handles about 7 per hour. For 80 articles that is roughly 11 hours of work. The eval workflow adds one cheap LLM call per article — on AirOps credits, a few dollars for the whole batch — plus 2 to 3 hours of editor time on the flagged set.
The break-even is around 30 articles per month. Below that, just read them.
Who this suits, and where it breaks down
Good fit: content teams shipping 50+ pieces a month with a fixed template. The rules are stable, so the eval stays accurate for months.
Poor fit: opinion pieces, thought leadership, or anything where “did this pass” is a judgement call. An LLM scorer will happily pass a technically-compliant article that is boring, and it will fail good writing for exceeding a word count by 40.
One more limitation worth knowing: AirOps does not give you per-rule scoring out of the box. You get whatever the eval prompt returns, which means your accuracy depends on how precisely you wrote the rules. Fuzzy rule wording is the number one reason these gates get switched off after a month.
Quick Summary
- Validate AirOps batches with a second, cheap eval workflow, not by reading every draft.
- Use 6-10 concrete rules and require JSON schema output so results are countable.
- Route by score bands (90+/60-89/<60), not by a single pass flag.
- One regeneration attempt recovers about half of failures; two or more is waste.
- Break-even versus manual review is roughly 30 articles per month.
- It does not work for opinion content where quality is a judgement call.
