Numbers here assume quantised inference, which is how nearly everyone runs these models in practice.

How much VRAM does each size need?

Rough rule: at 4-bit quantisation, a model needs about 0.6GB of VRAM per billion parameters, plus room for context. Context is the part people forget.

At 8-bit the numbers roughly double. At full precision they double again, which is why almost nobody does that outside research.

What hardware should you actually buy or rent?

For a laptop with unified memory, 24GB lets you run a mid-size model at 4-bit with an 8K context and reasonable speed. Below 16GB, stay in the small tier.

For desktops, a single RTX 4090 with 24GB is the sweet spot for 30–40B models. Two cards get you into 70B territory, but consumer cards usually lack the interconnect to scale cleanly — expect to split layers manually.

Renting is often cheaper than buying if your usage is bursty. A cloud GPU instance at hourly rates costs less than a $1,600 card you use two hours a week.

How do you configure the runtime?

1. Pick a loader that supports GGUF or a similar quantised format.
2. Set GPU layer offload to the maximum your VRAM allows; only offload the rest to CPU if forced.
3. Cap context at what you actually need. 8K is enough for a single file; 32K costs real memory.
4. Test throughput with a 500-word prompt and note tokens per second.
5. Save the config — good settings are easy to lose when you swap models.

If tokens per second falls below about 15, the model is too large for the hardware. Drop a tier.

What are the hidden costs?

Electricity and heat, mainly. A 300W card running inference at load for hours adds up, and a desktop under sustained load gets loud. For a home office, that is a real annoyance.

There is also maintenance. Updating loaders breaks quantisation compatibility more often than you would expect, and you own every failure. A hosted API moves that burden to someone else.

Self-host or use the API?

Self-host when privacy is non-negotiable, when your volume is high enough that per-token pricing beats hardware cost, or when you want to fine-tune. Use a hosted API when your usage is light, when you want zero maintenance, or when you need the largest model and cannot justify the hardware.

A reasonable middle path: develop and test against a hosted API, then move to local inference once you know the workflow is worth the machine.

Quick Summary

• At 4-bit, budget roughly 0.6GB VRAM per billion parameters plus context.
• 7–9B runs on 8GB; 30–40B wants 24GB; 70B+ needs multi-GPU.
• 8-bit roughly doubles VRAM; full precision is impractical outside research.
• Aim for 15+ tokens per second or drop a model tier.
• Rent GPUs for bursty use; buy only if you use it daily.

This guide covers choosing a size, running it, and the prompt habits that separate a helpful answer from generic boilerplate.

What is Qwen4?

Qwen4 is a family of language models from Alibaba’s Qwen team, released in multiple parameter sizes with open weights under a permissive licence. It handles code generation, refactoring, and explanation, and supports long context for reading through a repository. You can run it locally, or use it through a hosted API.

The open weights matter for privacy. Code you feed to a hosted model leaves your machine; code you feed to a local Qwen4 does not.

Which size should you run?

Match the model to your hardware, not to the leaderboard. A model you cannot run at usable speed is worse than a smaller one that responds instantly.

• Small variants (7–9B): run on 8GB of VRAM with quantisation. Good for autocomplete and short functions.
• Mid variants (30–40B): need roughly 24GB of VRAM at 4-bit. Handles refactoring and multi-file reasoning.
• Large variants (70B+): want multiple GPUs or a rented instance. Closest to frontier quality.

If you have a laptop with 16GB of unified memory, start at the mid tier with 4-bit quantisation and accept that very long context will be slow.

How do you set it up locally?

1. Install a runtime that handles quantised weights — most people use a GGUF-based loader.
2. Download the quantised weights matching your VRAM budget.
3. Set a context length of 8K to start; higher eats memory fast.
4. Test with a real file from your project, not a toy snippet.

Expect the first load to take a minute as weights move to GPU memory. Subsequent prompts are fast.

How do you prompt Qwen4 for code?

Ask for a diff, not a description. “Rewrite this function to remove the nested loop” gets you code. “How can I improve this function” gets you an essay about time complexity.

Three things that consistently improve results:

• Paste the surrounding code, not just the function — imports and types matter.
• State the target language version, e.g. Python 3.12 or ES2023.
• Ask for the full file when the change touches several functions; partial output breaks on copy-paste.

Where does it fall short?

Qwen4 is weaker than frontier models on novel algorithm design and on very large refactors spanning a hundred files. It also hallucinates library APIs occasionally, especially for fast-moving packages. Always run the code before trusting it.

The hosted API route also has regional availability quirks worth checking before you build a product on it.

Who is Qwen4 for?

Developers with a GPU who want a private, no-subscription coding assistant. Also anyone building a product who needs weights they can host themselves. If you want the absolute best reasoning on hard problems and do not care about privacy or cost, a closed frontier model still leads.

Quick Summary

• Qwen4 is Alibaba’s open-weight model family with strong coding performance.
• Choose size by VRAM: 7–9B on 8GB, 30–40B on 24GB, 70B+ on multiple GPUs.
• Run locally for privacy, or via hosted API for convenience.
• Ask for diffs and paste surrounding code for better output.
• Weaker on novel algorithms and huge refactors; verify library calls.