How to Use Qwen Code With Local Models

September 16, 2026

Qwen Code speaks the OpenAI-compatible API format, which means you can point it at a model running on your own hardware instead of Alibaba’s servers. That is the feature that matters if your code cannot leave the building.

The setup is one environment variable. The hard part is picking a local model large enough to drive an agent loop without stalling.

Why Would You Run a Coding Agent Against a Local Model?

Three reasons: source code confidentiality, no per-request cost, and no daily quota. Hospitals, defense contractors, and anyone under a strict data agreement usually cannot send repositories to a hosted API at all.

Can Qwen Code Use Ollama or vLLM?

Yes. Because it accepts any OpenAI-compatible endpoint, both work. Ollama exposes http://localhost:11434/v1 and vLLM serves the same shape on whatever port you launch it with.

Point the CLI at your server with the base URL and a placeholder key, then select the model name you loaded.

How Do You Configure It?

Start your server: ollama serve or vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct
Set OPENAI_BASE_URL=http://localhost:11434/v1
Set OPENAI_API_KEY=local (any non-empty string)
Set OPENAI_MODEL=qwen3-coder to match the loaded model
Launch qwen and verify it responds

Which Local Model Should You Load?

Agentic coding needs tool-calling and long context. Below roughly 30 billion parameters, agents get confused about when to call tools and start hallucinating file contents.

Best fit: Qwen3-Coder 30B-A3B — MoE, fast on a single 24GB GPU
Quality first: Qwen3-Coder 480B if you have the VRAM or a multi-GPU box
Avoid: 7B chat models; they patch the wrong files

What Breaks in Practice?

Two things. First, context length. Your local server may cap at 8K tokens while the agent assumes far more, so long files get truncated and edits land in the wrong place. Raise the server context window explicitly. Second, speed — a 480B model on one GPU can take thirty seconds per tool call, which makes an interactive loop painful.

Quick Summary

Qwen Code works with any OpenAI-compatible endpoint, including Ollama and vLLM.
Set the base URL, a dummy key, and the model name.
Pick 30B or larger for reliable tool calling.
Raise server context length or edits will truncate.
Local models remove quota limits but cost GPU time.

Tags:

qwen-code