How to Quantize GPT-OSS for Low-VRAM GPUs

Running GPT-OSS at full precision needs far more VRAM than most people own. Quantising it to GGUF shrinks the 20B model enough to fit an 8GB or 12GB card, and the quality loss at 4-bit is smaller than you would expect.

What does quantisation actually do to GPT-OSS?

Quantisation stores the model weights with fewer bits per value. Instead of 16-bit floats you get 4 or 5 bit integers plus a scale factor. A 20B model drops from roughly 40GB of weights to about 11-13GB at Q4_K_M. Inference gets faster too, because memory bandwidth is usually the bottleneck, not raw compute.

Which GGUF quant should you pick?

Quant	Approx size	VRAM needed	Quality vs FP16
Q8_0	~21GB	24GB	Near lossless
Q6_K	~16GB	20GB	Very slight loss
Q5_K_M	~14GB	16GB	Barely noticeable
Q4_K_M	~12GB	12GB	Good default
Q3_K_M	~9.5GB	10GB	Noticeable on reasoning

Q4_K_M is the sane default. I would not go below Q3_K_M — factual recall and multi-step arithmetic start falling apart, and you paid for a 20B model only to get something closer to a 7B in behaviour.

How do you serve a quantised GPT-OSS as an API?

1. Download a GGUF build from a community repo on Hugging Face.
2. Start llama.cpp’s server: ./llama-server -m gpt-oss-20b-Q4_K_M.gguf -c 8192 -ngl 99 --port 8080.
3. Point any OpenAI-compatible client at http://localhost:8080/v1.
4. Set -ngl 99 to offload every layer to the GPU; drop it if you run out of VRAM.

The context window matters. At 8192 tokens a 12GB card is fine; push to 32K and the KV cache eats several extra gigabytes, which can force layers back onto the CPU and cut speed by half.

What breaks at low precision?

Code generation holds up well — most users will not notice. Long chains of logic degrade first, and the model becomes more likely to repeat a phrase when the prompt is ambiguous. If you run agentic loops with tool calls, test at your target quant before shipping; a 4-bit model sometimes malforms JSON arguments where the 8-bit version would not.

When is quantisation the wrong move?

If you have a 48GB or 80GB card, just run Q8_0 or the unquantised weights. The speed gain from quantisation mostly disappears once you are not memory-starved, and you avoid the quality gamble entirely. Also skip aggressive quants for anything safety-critical.

Key Takeaways

• Q4_K_M fits GPT-OSS 20B on a 12GB GPU at around 12GB of weights.
• Do not drop below Q3_K_M; reasoning quality falls off a cliff.
• llama-server exposes an OpenAI-compatible endpoint on localhost.
• Watch the KV cache — long contexts can undo your VRAM savings.
• Below 48GB of VRAM, quantisation is usually worth it; above it, skip it.