How to Self-Host Mistral Codestral 2 for Private Code AI

You can self-host Mistral Codestral 2 on a single GPU and keep every line of code inside your network. A 24GB card handles the model at useful speed for a small team, and there’s no per-token bill.

Lots of companies ban cloud AI coding tools because source code is the crown jewels. That ban usually means no autocomplete at all. It doesn’t have to — an internal endpoint removes the data-leak objection entirely.

What Hardware Do You Need?

• Minimum: a 16-24GB GPU for the model at 4-bit or 8-bit quantization.
• Practical: a 24GB card serves 3-8 developers with sub-second completions.
• Throughput: add a second GPU or use a smaller quantized build for teams beyond ten.

Completion requests are short, so throughput is more about concurrency than raw compute. Batching is your friend.

How Do You Deploy It?

1. Pull the weights and run them under vLLM or Ollama — both expose an OpenAI-compatible endpoint.
2. Start the server with a modest max-model-len; code completion doesn’t need a 128k context.
3. Enable prefix caching so repeated file context isn’t reprocessed every keystroke.
4. Point your editor extension’s base URL at the internal server, and set a dummy API key.
5. Put it behind your internal auth so only developers can reach it.

How Much Does It Cost to Run?

Upfront hardware is the main cost, or roughly $300-$600 a month for a rented 24GB-class GPU. Against per-seat cloud subscriptions at $10-$30 per developer per month, break-even lands around 15-20 developers.

Below that headcount the API is cheaper. The case for self-hosting below break-even is compliance, not money — and for regulated code that’s often reason enough.

What Gets Harder When You Self-Host?

You own uptime and updates. A crashed completion server is invisible until the whole team notices their editor has gone quiet, so add a health check. Quantization also costs a little accuracy compared to the hosted model — usually invisible in completion, occasionally noticeable on complex suggestions.

Latency needs tuning. If the model is behind your VPN and a developer is remote, round trips add up. Run it near the team, not in a distant data center.

Who Should Do This?

Companies with strict data policies, defense and health-adjacent teams, and anyone with existing GPU capacity. If you’re an individual developer with no compliance constraint, the hosted API is simpler and cheaper — don’t self-host out of principle.

Quick Summary

• Codestral 2 self-hosts on a 16-24GB GPU with no per-token cost.
• One 24GB card serves 3-8 developers; use vLLM or Ollama with prefix caching.
• Break-even versus cloud seats is around 15-20 developers.
• Worth it for compliance and existing GPUs, overkill for solo devs.