How to Run Llama or Qwen Inference on Cerebras

November 21, 2026

Cerebras makes wafer-scale chips and sells inference on them through an API. The pitch is speed: its chat completions API is OpenAI-compatible and commonly reaches thousands of output tokens per second on supported Llama and Qwen models, far above GPU serving speeds.

For most developers, the integration is a base URL change. The interesting work is knowing what to expect from the limits.

How do you make your first call?

Create an API key, then point an OpenAI client at the Cerebras base URL and use a model ID from their supported list.

client = OpenAI(base_url="https://api.cerebras.ai/v1", api_key=KEY)
resp = client.chat.completions.create(model="llama-3.3-70b", messages=[...])

Because the schema matches, streaming, tool calls and JSON mode work with existing client code. Model IDs differ from OpenAI’s, so check the current list rather than guessing — the supported set changes as new models are added.

Which models are actually available?

Mid-sized open models are the sweet spot: Llama 3.3 70B, Llama 3.1 8B, Qwen variants, and a few others depending on the month. Frontier proprietary models are not served here.

If your product depends on a specific model, verify it is on the list before designing around it. That single check prevents most wasted weeks.

What are the real limits?

Two that matter:

Context length. Supported context is generous but not unlimited; long-document workloads need checking per model.

Rate limits. Free and lower tiers are limited in requests per minute and tokens per minute. The speed per request is exceptional, but you can hit concurrency ceilings before you hit any latency problem. For a batch job, queue and throttle rather than firing 500 parallel requests.

When does the speed actually help?

It helps when a human is waiting on a stream. A 1,000-token answer at high tokens per second finishes in a second or two, which changes the feel of a chat product more than any UI polish.

It helps less when your bottleneck is a tool call to a slow external API, or when you are batching overnight jobs — the job finishes in four minutes instead of forty, which is nice and not transformative.

It does not help reasoning quality. Faster tokens do not make a weaker model smarter; chain-of-thought still takes as many tokens as it takes.

What does it cost?

Priced per million tokens, with a free tier for development and usage-based pricing above. Per-token prices on these open models are competitive with GPU providers, and the differentiator is throughput rather than price.

Watch the spend shape: because responses are fast, it is easy to run ten times more evaluations than planned. Set a spending cap in the dashboard before running any benchmark sweep.

Where it is the wrong choice

Teams needing GPT-class or Claude-class reasoning. Those models are not on the platform.

Fine-tuning workflows. This is an inference service; if you need to train, that is a different provider.

Workloads with bursty concurrency beyond plan ceilings. A sudden traffic spike will hit rate limits, and your fallback path needs to exist before that spike.

Quick Summary

  • The API is OpenAI-compatible: change base URL and model ID, keep your client code.
  • Available models are open-weight mid-size models (Llama 3.3 70B, Llama 3.1 8B, Qwen variants) — no frontier proprietary models.
  • High per-request speed, but rate limits cap concurrency on lower tiers; queue batch work.
  • Biggest visible win is streaming to a waiting human; overnight batches gain less.
  • Priced per million tokens with a free development tier; set a spending cap before benchmark sweeps.