How to Benchmark and Plan Around Cerebras Throughput
Cerebras publishes impressive throughput figures, and they are real under the right conditions. Your conditions will be different. A benchmark from the vendor measures a single request at low concurrency; your product measures 200 users hitting the same endpoint at 9am.
This is how to measure both and turn the results into a capacity plan.
What should you measure?
Four numbers, per model, per concurrency level:
- Time to first token — what a streaming user perceives as responsiveness.
- Output tokens per second — how fast the answer completes.
- Requests per minute before errors — where rate limiting begins.
- p95 latency at fixed concurrency — the number you put in an SLO.
Medians are misleading. Collect at least 100 requests per configuration.
How do you run a fair benchmark?
Hold everything constant except concurrency:
- Same prompt, same max_tokens, same temperature, same model across runs.
- Vary prompts too, in a second sweep — a 10-token prompt and a 2,000-token prompt load the system differently.
- Run concurrency steps of 1, 5, 20 and 50, and record errors as data rather than retrying.
- Measure from the client, not the server, and include network.
Run each step three times on different hours. A single afternoon’s numbers include whatever else was happening on the shared endpoint.
What do the numbers look like in practice?
Single-request speed on mid-size models is high — often in the thousands of tokens per second. At concurrency of 20 or 50, per-request throughput drops and p95 latency climbs, and somewhere on that curve you hit rate limits rather than compute limits.
The shape matters more than absolute values: if p95 latency rises 4× between concurrency 5 and 20, your design needs queuing rather than more parallelism.
How does this change your architecture?
Three decisions it should influence:
Fallback provider. Rate limits are per plan; a spike needs a second endpoint. Build the fallback before launch and test it under load, since an untested fallback usually fails on schema differences (different tool-call format, different stop behaviour).
Queueing instead of retries. When you hit a 429, retry with exponential backoff plus jitter, and cap total waits. For batch work, a queue with a fixed concurrency worker pool is more predictable than a burst.
Streaming as the default. If time to first token is under a few hundred milliseconds, stream; users read while it generates and the perceived speed gain is larger than the measured gain.
What does benchmarking cost?
A sweep of four concurrency levels times three repeats on 100 requests of 500 tokens is roughly 600,000 output tokens — small at per-million-token pricing, but not zero, and it multiplies if you repeat for five models. Budget a few dollars, and set a hard spending cap so a looping script cannot run all night.
What the benchmark will not tell you
Quality. Tokens per second says nothing about whether the model is good enough for your task; that needs an eval set.
Cost per successful answer. Failed or retried calls inflate real cost above the sticker price. Measure cost per completed task, not per token.
Provider incident days. Re-run the benchmark monthly; capacity changes as the provider adds or removes capacity.
Quick Summary
- Measure time to first token, output tokens per second, requests per minute to first error, and p95 latency.
- Sweep concurrency at 1, 5, 20 and 50 with identical prompts, three repeats, from the client side.
- Per-request speed is high; p95 latency and rate limits are what your capacity plan must respect.
- Build and load-test a fallback provider before launch — rate limits are per plan.
- Track cost per completed task, not per token, and re-benchmark monthly.
