How to Test an AI API with Postman Collections

LLM endpoints break the normal REST testing assumptions. Responses are slow, expensive, and sometimes streamed, which means the tests that work for a CRUD API do not carry over.

Direct answer: to test an AI API in Postman, create a collection with an environment holding your API key and base URL, add a Bearer {{token}} header, and use the Tests tab to assert on responseTime, status code, and the shape of the JSON. For streaming endpoints, disable the stream flag first — Postman’s test scripts run after the body is fully received.

How do you store the API key safely?

In an environment variable, never in the collection. Collections get exported and shared; environments can be marked secret per variable.

Create an environment called ai-dev with three variables:

• baseUrl = https://api.openai.com/v1
• apiKey = your key, marked as secret type
• model = gpt-4o-mini

Then use {{baseUrl}}/chat/completions and a header Authorization: Bearer {{apiKey}}. The secret type masks the value in the UI and redacts it from exported environments.

Commit the environment without values to a repo (Postman supports this), and pull real values from a vault or CI secret. Otherwise the first shared export leaks your key.

What should a basic assertion look like?

Go to the Tests tab and write plain JavaScript. pm is the global helper.

pm.test("status is 200", () => pm.response.to.have.status(200));
pm.test("responds under 8s", () => pm.expect(pm.response.responseTime).to.be.below(8000));
const body = pm.response.json();
pm.test("has a completion", () => {
  pm.expect(body.choices).to.be.an("array").that.is.not.empty;
  pm.expect(body.choices[0].message.content).to.be.a("string");
});
pm.test("token usage recorded", () => pm.expect(body.usage.total_tokens).to.be.above(0));

The usage assertion is the one people skip. It catches the case where a proxy or gateway strips the usage object, which silently breaks your billing estimates.

Do not assert on exact completion text. Models are non-deterministic even at temperature 0 across versions. Assert on structure, length bands, and the presence of required fields.

How do you chain requests?

Set a variable from a response, use it in the next request’s auth header or body:

const j = pm.response.json();
pm.collectionVariables.set("threadId", j.id);
pm.test("id captured", () => pm.expect(j.id).to.be.a("string"));

Then reference {{threadId}} downstream. This is how you test the assistant-thread pattern: create a thread, post a message, run it, fetch the reply, delete the thread. Five requests, one run.

Use collection variables for values shared across the run, environment variables for values that change between staging and production. Mixing them causes the classic bug where a test passes locally and hits production URLs in CI.

What about streaming responses?

Postman buffers the body before running tests, so stream: true gives you a wall of data: {...} chunks that fail JSON parsing. Two options:

1. Keep a second request with stream: false for assertions, and test the streaming one manually.
2. Assert on the raw text: pm.expect(pm.response.text()).to.include("data:") and check the terminal [DONE] marker.

Option one is more useful. Streaming is a transport detail; the content assertions belong on the non-streaming call.

The free plan also caps request timeout, and long generations on a big model can exceed it. Set a longer timeout in request settings under Timeout, up to 0 in some versions meaning unlimited — but a test that takes 90 seconds will make the collection runner painful.

How do you run it in CI?

Newman, Postman’s CLI runner:

npm install -g newman
newman run ai.postman_collection.json -e ai-dev.postman_environment.json --reporters cli,junit

Export the collection and a value-free environment, and inject secrets through environment variables that Newman reads with --env-var. In GitHub Actions that means one step, and the JUnit reporter plugs into whatever test-viewer you already have.

Cost note: a collection with 15 LLM requests run on every push to a busy repo adds up. Gate it to nightly builds and pull requests touching prompt templates, not every commit.

Which parts are genuinely weak

Mock servers do not work for LLM endpoints — the whole point is non-deterministic output, so a mock proves nothing about the real model.

Postman is also heavy for this job. If your team lives in the terminal, a 40-line Python script with httpx and pytest gives you the same coverage with proper fixtures and no GUI. Postman’s advantage is the shared workspace and the ability to hand a collection to a non-engineer.

Finally, Postman cannot tell you whether an answer is correct, only whether it is well-formed. Quality evaluation is a separate discipline — use a labelled set and a grader, not assertions.

Key Takeaways

• Keep keys in secret-typed environment variables, never inline in a collection.
• Assert on status, latency, response shape, and usage.total_tokens — never on exact text.
• Chain requests with pm.collectionVariables.set() to test multi-turn flows.
• Postman buffers streamed bodies; test content on a stream: false twin request.
• Run collections in CI with Newman, and gate them to nightly or prompt-touching PRs to control cost.