How to Monitor LLM Apps in Arize AI Without a Data Team

November 21, 2026

Arize AI is an evaluation and observability platform for machine learning and LLM applications. You send it traces of your model calls, it groups them into dashboards, and it can score outputs against rubrics so you see quality drift before users complain.

The usual objection is that this needs a data team. It does not — a single backend engineer can get useful signal in a day. Here is the minimum viable setup.

What should you send first?

Traces with inputs, outputs, latency and model name. That is it. Skip embeddings, skip span hierarchies, skip custom metrics for week one.

One trace per user request, with a child span per model call, gives you the three numbers that matter: p95 latency, error rate, and cost per request. Most LLM quality regressions show up as one of those three changing first.

Tag every trace with your app version and prompt version. Without prompt version tags, you cannot tell whether a quality drop came from a code deploy or a prompt edit — and prompt edits are the more common cause.

How do you score quality when nobody will hand-label?

Use an LLM as judge, with a rubric short enough to be reliable. Three criteria at most to start: does the answer address the question, is every claim supported by the retrieved context, and is the format what the UI expects. Rate 1–5.

Judge every trace for the first 500 requests to calibrate, then sample 5 percent. Calibrate by hand-labeling about 50 traces yourself and comparing to the judge. If the judge agrees on fewer than 75 of those 50 (adjusting expectations for a small sample), tighten the rubric before trusting the score.

The cost is real but small: judging 5 percent of traffic with a mid-tier model usually amounts to a few dollars a month for a 10k-request-a-week app.

What do you monitor after the dashboard exists?

  1. Groundedness — share of answers where every claim traces to a retrieved document.
  2. Refusal rate — sudden drops mean the prompt got more permissive.
  3. Output length distribution — a jump often means the model started padding.
  4. Retrieval hit rate — how often the retrieved chunks contained the answer at all.
  5. Cost per conversation, not per call.

Set alerts on the first and last. Groundedness below your baseline by 10 points is the signal that something broke upstream, usually an index rebuild or a chunking change.

What does Arize not do for you?

It does not fix prompts, and its dashboards need someone to read them. A monitoring tool with no weekly review is a very expensive log store.

Sampling traces at 100 percent for high-volume apps gets expensive in storage; most teams settle at 10 to 20 percent after the first month once confidence is established.

Judge models drift. If you upgrade the judge’s own model, your scores shift and your history becomes uncomparable. Pin the judge version and note the date of any change.

Who should use it

Teams running an LLM feature in production with more than a handful of users, especially anything retrieval-based where groundedness can quietly degrade. Companies already using Arize for classic ML models get the most value, since the alerting habits and account already exist.

It is unnecessary for prototypes and for products where outputs are cheap to check by eye, such as internal summarisation for a ten-person team.

Quick Summary

  • Start with traces containing input, output, latency and model — plus prompt-version tags — before adding anything else.
  • Use a 3-criteria LLM judge on 100% of the first 500 requests, then sample 5%.
  • Alert on groundedness drops and cost per conversation; review weekly or the dashboard is dead weight.
  • Judge model upgrades reset your score history — pin the version.
  • Free and lower tiers exist; cost scales with trace volume and evaluation runs.