How to Self-Host Qwen4 Locally: Hardware and Setup Guide
Running Qwen4 on your own machine is the whole point of an open-weight model — no API keys, no per-token billing, no code leaving your network. The trade-off is hardware. This guide maps model size to real VRAM requirements so you buy or rent the right thing.
Numbers here assume quantised inference, which is how nearly everyone runs these models in practice.
How much VRAM does each size need?
Rough rule: at 4-bit quantisation, a model needs about 0.6GB of VRAM per billion parameters, plus room for context. Context is the part people forget.
| Model size | Weights (4-bit) | With 8K context | Runs on |
|---|---|---|---|
| 7–9B | ~5GB | ~7GB | RTX 3060 12GB |
| 30–40B | ~20GB | ~24GB | RTX 4090 / 3090 |
| 70B+ | ~40GB | ~48GB | 2× 24GB or A100 |
At 8-bit the numbers roughly double. At full precision they double again, which is why almost nobody does that outside research.
What hardware should you actually buy or rent?
For a laptop with unified memory, 24GB lets you run a mid-size model at 4-bit with an 8K context and reasonable speed. Below 16GB, stay in the small tier.
For desktops, a single RTX 4090 with 24GB is the sweet spot for 30–40B models. Two cards get you into 70B territory, but consumer cards usually lack the interconnect to scale cleanly — expect to split layers manually.
Renting is often cheaper than buying if your usage is bursty. A cloud GPU instance at hourly rates costs less than a $1,600 card you use two hours a week.
How do you configure the runtime?
- Pick a loader that supports GGUF or a similar quantised format.
- Set GPU layer offload to the maximum your VRAM allows; only offload the rest to CPU if forced.
- Cap context at what you actually need. 8K is enough for a single file; 32K costs real memory.
- Test throughput with a 500-word prompt and note tokens per second.
- Save the config — good settings are easy to lose when you swap models.
If tokens per second falls below about 15, the model is too large for the hardware. Drop a tier.
What are the hidden costs?
Electricity and heat, mainly. A 300W card running inference at load for hours adds up, and a desktop under sustained load gets loud. For a home office, that is a real annoyance.
There is also maintenance. Updating loaders breaks quantisation compatibility more often than you would expect, and you own every failure. A hosted API moves that burden to someone else.
Self-host or use the API?
Self-host when privacy is non-negotiable, when your volume is high enough that per-token pricing beats hardware cost, or when you want to fine-tune. Use a hosted API when your usage is light, when you want zero maintenance, or when you need the largest model and cannot justify the hardware.
A reasonable middle path: develop and test against a hosted API, then move to local inference once you know the workflow is worth the machine.
Quick Summary
- At 4-bit, budget roughly 0.6GB VRAM per billion parameters plus context.
- 7–9B runs on 8GB; 30–40B wants 24GB; 70B+ needs multi-GPU.
- 8-bit roughly doubles VRAM; full precision is impractical outside research.
- Aim for 15+ tokens per second or drop a model tier.
- Rent GPUs for bursty use; buy only if you use it daily.
