Most agent projects die at the same place: the demo works, the prompt gets tweaked for one edge case, and three other cases quietly break. Nobody notices until a customer does. Evals are the boring discipline that prevents this, and in OpenAI AgentKit they are built into the same workspace as the agent itself.

What should an eval contain?

An input, an expected behavior, and a grading rule. The grading rule can be exact match, a substring check, a model-based grader with a rubric, or a custom function.

The key insight is that most agent outputs are not exactly matchable. “Did the agent correctly refund the customer” has no single correct string. That is where model-based graders with a written rubric earn their place.

How do you set up your first eval run?

1. Collect 15-30 real inputs from actual usage or realistic guesses.
2. Write the expected behavior for each in one sentence.
3. Choose a grader: exact, contains, model-graded, or custom.
4. Run the eval against your current agent and record the pass rate.
5. Change one thing in the prompt, rerun, compare pass rates.

That last step is the whole point. An eval suite without a before-and-after comparison is just a smoke test.

How much do evals cost?

Every eval run calls a model, and a model-based grader means two calls per case: one for the agent, one for the grader. Thirty cases with a model grader is roughly sixty model calls per run. Run that on every prompt edit and the token spend adds up, so trigger full evals on meaningful changes and keep a smaller smoke set for quick iterations.

How many test cases is enough?

Fifteen to thirty covers a lot for an internal tool. Below ten you are mostly testing the happy path. Above fifty, maintenance becomes a job of its own and stale cases start producing false alarms.

Deliberately include failure cases: inputs that should trigger a refusal, requests outside the agent’s scope, and ambiguous prompts. Agents fail at boundaries far more often than in the middle.

What are the limits of AgentKit evals?

They measure what you thought to test. A good score on twenty cases does not mean the agent will behave on the thousandth real input. Also, model-based grading is itself a model judgment and can be inconsistent run to run; for high-stakes checks, prefer deterministic assertions.

Teams shipping customer-facing agents should treat this as a regression net, not a certification. Teams still prototyping can skip evals entirely and add them once the design stabilizes.

Quick Summary

• AgentKit evals define inputs, expected behavior and grading rules.
• Use deterministic graders when possible; model graders for open-ended output.
• 15-30 cases is a practical starting suite.
• Each run consumes tokens; model grading doubles the calls.
• Evals catch regressions, not unknown failure modes.

The interesting design choice is that AgentKit does not force you to pick code or no-code. You can wire a flow visually and then drop into code for the parts that need logic. For teams where a PM and an engineer both touch the same agent, that shared surface is the main selling point of OpenAI AgentKit.

What is in the box?

Four pieces. Agent Builder is a canvas where you drag nodes for model calls, tools, and branching logic. The Agents SDK is the code-first path, with multi-agent handoffs. Connector Registry exposes third-party data sources as tools. Evals let you test agent output against defined criteria.

Evals are the underrated one. Without them you are guessing whether a prompt change actually improved anything.

How do you build your first agent?

1. Open Agent Builder in the OpenAI platform dashboard.
2. Add a start node and a model node; pick a model like GPT-4o or a newer release.
3. Write the agent instructions in the model node — purpose, tone, boundaries.
4. Attach one tool, such as a web search or a connector.
5. Run a test prompt in the preview panel, then export the workflow or wire it to the SDK.

One tool again. Multi-tool agents fail in ways that are painful to debug on day one.

How much does AgentKit cost?

There is no separate AgentKit fee. You pay standard OpenAI API token rates for the model calls your agent makes. A small agent using a mid-tier model costs fractions of a cent per interaction; agent loops with long context can reach several cents. The eval runs also consume tokens, which surprises people who run them on every commit.

Should you use the visual builder or the SDK?

Prototype visually, then port to the SDK before you ship. Visual flows are hard to review in a pull request.

What are the limitations?

You are inside OpenAI’s ecosystem. Model choice means OpenAI models, and connectors are the ones supported by the registry rather than anything you can imagine. Teams committed to a multi-vendor model strategy will find that constraining.

Also, long-running agents with many steps get expensive quickly, and there is no built-in cost guardrail. You add spend limits at the account level.

Suits product teams wanting to ship an internal assistant fast with evaluation built in. Less suited to teams already deep in another framework who would be duplicating orchestration.

Quick Summary

• AgentKit bundles Agent Builder, Agents SDK, Connector Registry and evals.
• No extra fee; you pay standard OpenAI token rates.
• Start with one tool and clear instructions, then expand.
• Prototype in the visual builder, port to the SDK for production.
• OpenAI-model-only and no built-in spend limits.