How to Self-Host Kimi K2 Instruct With vLLM

Pulling the weights for Kimi K2 Instruct costs nothing. Serving it is the expensive part. This is a realistic look at what self-hosting a large mixture-of-experts model takes, and when the API is simply the better deal.

Why self-host a model this big?

Three reasons hold up. Data never leaves your network. You avoid per-token pricing at high volume. And you keep full control over versioning — a hosted model can be updated under you overnight, breaking prompts you had tuned.

Everything else is a reason not to: hardware cost, operations load, and the fact that you become the on-call engineer for a GPU cluster.

What hardware do you actually need?

Setup	VRAM	Notes
FP8, full quality	~1TB	Multi-node, datacenter only
INT4 quantised	~250-350GB	4x 80GB GPUs, feasible
Smaller quantised	~150GB	2x 80GB, quality loss grows

Because it is MoE, inactive experts still occupy memory even though they do not compute. You cannot squeeze it the way you would a dense model of similar active size. Budget for the whole set of weights, not the active slice.

How do you serve it with vLLM?

1. Install vLLM with pip install vllm and confirm your CUDA version matches the wheel.
2. Download the quantised weights from Hugging Face into a shared directory.
3. Launch with tensor parallelism across your GPUs: vllm serve moonshotai/Kimi-K2-Instruct --tensor-parallel-size 4 --max-model-len 32768.
4. Expose the OpenAI-compatible endpoint and point your client at it.
5. Cap max-model-len — the KV cache for very long contexts can exhaust VRAM and crash the server.

Where does self-hosting go wrong?

Cold starts are brutal. Loading hundreds of gigabytes from disk takes minutes, so autoscaling on demand does not work — you must keep instances warm, which means paying 24/7. Quantisation often looks fine in benchmarks and then degrades on your specific task, so always test before committing.

Throughput is the other trap. A single user query might feel fine. Ten concurrent users can queue badly if tensor parallelism and batching are not tuned, and tuning MoE throughput is a genuine skill, not a config flag.

When is the API the smarter choice?

Almost always, unless one of the three reasons above is decisive. Do a break-even calculation: the monthly cost of reserved GPUs versus your current token spend. For most teams below a billion tokens a month, the GPUs cost more and come with real operational risk.

Self-host when data rules forbid the API, when you cross high volume consistently, or when version stability is a hard requirement. Otherwise, rent the model and spend your engineering time on the product.

Key Takeaways

• Kimi K2 Instruct weights are free but need roughly 250GB+ VRAM when quantised.
• MoE inactive experts still consume memory, so small quantisation does not help much.
• vLLM serves it via an OpenAI-compatible endpoint with tensor parallelism.
• Always-on GPU cost and slow cold starts undercut any autoscaling plan.
• Self-host for privacy or scale; otherwise the API is cheaper and safer.