What does quantisation actually do to GPT-OSS?

Quantisation stores the model weights with fewer bits per value. Instead of 16-bit floats you get 4 or 5 bit integers plus a scale factor. A 20B model drops from roughly 40GB of weights to about 11-13GB at Q4_K_M. Inference gets faster too, because memory bandwidth is usually the bottleneck, not raw compute.

Which GGUF quant should you pick?

Q4_K_M is the sane default. I would not go below Q3_K_M — factual recall and multi-step arithmetic start falling apart, and you paid for a 20B model only to get something closer to a 7B in behaviour.

How do you serve a quantised GPT-OSS as an API?

1. Download a GGUF build from a community repo on Hugging Face.
2. Start llama.cpp’s server: ./llama-server -m gpt-oss-20b-Q4_K_M.gguf -c 8192 -ngl 99 --port 8080.
3. Point any OpenAI-compatible client at http://localhost:8080/v1.
4. Set -ngl 99 to offload every layer to the GPU; drop it if you run out of VRAM.

The context window matters. At 8192 tokens a 12GB card is fine; push to 32K and the KV cache eats several extra gigabytes, which can force layers back onto the CPU and cut speed by half.

What breaks at low precision?

Code generation holds up well — most users will not notice. Long chains of logic degrade first, and the model becomes more likely to repeat a phrase when the prompt is ambiguous. If you run agentic loops with tool calls, test at your target quant before shipping; a 4-bit model sometimes malforms JSON arguments where the 8-bit version would not.

When is quantisation the wrong move?

If you have a 48GB or 80GB card, just run Q8_0 or the unquantised weights. The speed gain from quantisation mostly disappears once you are not memory-starved, and you avoid the quality gamble entirely. Also skip aggressive quants for anything safety-critical.

Key Takeaways

• Q4_K_M fits GPT-OSS 20B on a 12GB GPU at around 12GB of weights.
• Do not drop below Q3_K_M; reasoning quality falls off a cliff.
• llama-server exposes an OpenAI-compatible endpoint on localhost.
• Watch the KV cache — long contexts can undo your VRAM savings.
• Below 48GB of VRAM, quantisation is usually worth it; above it, skip it.

What is GPT-OSS and what sizes does it come in?

GPT-OSS ships in two main variants: a 20B parameter model and a larger 120B one. Both use a mixture-of-experts design, which means only a slice of the network fires on each token. That keeps the 20B model usable on a single 16GB GPU or a 32GB Mac, while the 120B version realistically wants 80GB+ of VRAM or a beefy multi-GPU rig.

Weights are on Hugging Face under a permissive license, so commercial use is allowed. The catch: no official fine-tuning recipe is bundled, and the safety tuning is lighter than the hosted OpenAI models.

How do you install GPT-OSS?

The fastest route is Ollama, then llama.cpp if you want full control.

1. Install Ollama: curl -fsSL https://ollama.com/install.sh | sh on Linux or macOS, or grab the Windows installer.
2. Pull the small model: ollama pull gpt-oss:20b. Expect about 13GB of download.
3. Run it: ollama run gpt-oss:20b and type a prompt.

On the 20B model I get roughly 35-45 tokens per second on an RTX 4090 at 4-bit quantisation. A 2020 MacBook Air with 8GB will not run it — you need at least 16GB of unified memory, and 32GB feels comfortable.

Is GPT-OSS free to use?

Yes. The weights cost nothing and there is no per-token billing. Your real cost is electricity and hardware. That is the whole appeal for teams who cannot send data to a third-party API — legal, medical and defence shops especially.

What you give up: no usage dashboard, no autoscaling, and you own every outage. If your GPU driver updates and breaks CUDA, that is your problem at 2am.

How does GPT-OSS compare with hosted models?

Who should run GPT-OSS and who should skip it?

It fits you if data residency matters, if you hit API rate limits constantly, or if you want to experiment without a bill running. Skip it if you need the strongest reasoning available or multi-modal input — the open weights lag the frontier models on hard math and long-context tasks, and there is no vision input here.

Key Takeaways

• GPT-OSS is OpenAI’s free open-weight model, downloadable and runnable offline.
• The 20B variant runs on 16GB VRAM; the 120B wants 80GB+.
• Ollama gives you a working local chat in about 15 minutes.
• No API cost, but you supply the hardware and the uptime.
• Not the right pick if you need true frontier reasoning or vision.