How to Price Self-Hosted LLM Inference per Request
Teams quoting “our GPU costs $2 an hour” are quoting the wrong number. The number that decides whether self-hosting makes sense is cost per thousand tokens at your actual utilisation, and it varies by 20x between a busy card and an idle one.
Direct answer: to price self-hosted Baseten inference per request, divide the GPU hourly rate by the tokens the card produces per hour, then multiply by your tokens per request. Add 15-25% for idle capacity, restarts, and headroom. Below about 50% utilisation, a hosted API at $0.15 per million input tokens is usually cheaper.
What are the inputs?
Four numbers, and you probably only have one of them.
- GPU hourly cost. Baseten Pro runs roughly $0.6-0.8/hr for an L4, $1.5-2/hr for an A10G, $4-6/hr for an H100. Prices move; read your own invoice.
- Throughput. Tokens per second the model produces at your batch size. An 8B model on an L4 with batching hits roughly 800-1,500 output tokens/sec.
- Request shape. Input tokens plus output tokens, averaged over real traffic.
- Utilisation. Fraction of wall-clock time the GPU is doing work.
You measure 2 with a load test, 3 from logs, and 4 from your autoscaler metrics. Guessing any of them produces a number that is wrong by an order of magnitude.
What is the formula?
Straightforward once you have the four terms:
price_per_hour = gpu_hourly_cost / utilisation
cost_per_request = (price_per_hour / 3600) * seconds_per_request
cost_per_1M_tokens = price_per_hour * 1e6 / (tokens_per_second * 3600)
Worked example. An L4 at $0.70/hr, 60% utilisation, so effective $1.17/hr. At 1,000 output tokens/sec that is 3.6M tokens/hr, giving $0.32 per million output tokens.
Compare with a hosted small model at around $0.60 per million output tokens. Self-hosting is half the price — but that assumes 60% utilisation, which assumes steady traffic.
Why does utilisation dominate everything?
Because the GPU bills whether or not it is answering.
| Utilisation | Effective $/hr (L4) | $ per 1M output tokens |
|---|---|---|
| 10% | $7.00 | $1.94 |
| 30% | $2.33 | $0.65 |
| 60% | $1.17 | $0.32 |
| 90% | $0.78 | $0.22 |
At 10% utilisation you are paying 6x the hosted API rate for the privilege of running your own GPU. This is the trap teams fall into: they compute cost at full throughput, which is what the benchmark measures, and ship into production with 8% utilisation.
What else belongs in the number?
Engineering time, and it is not small. Serving infra needs versioning, health checks, failure handling, and someone on call. Budget 0.2-0.5 FTE for a production deployment. At a loaded $150k/year that is $30k-75k annually, which at 3.6M tokens/hour is irrelevant and at 200k tokens/day is the entire cost.
Also add: storage for weights (a 70B in fp16 is 140 GB, billed monthly), egress on responses, and the evaluation cost of validating every model upgrade.
What you do not add is the price you would have paid a hosted vendor — that is the comparison, not an input.
When does self-hosting actually win?
Three situations, and they are narrower than the marketing suggests.
Very high sustained volume. Above roughly 200M tokens a month, utilisation stays high and the per-token gap compounds.
Fine-tuned models. If a 3B model tuned on your data matches a 70B general model, you are comparing a 3B’s serving cost with a 70B’s API price. That is a 10-20x gap and the only case where the economics are dramatic.
Latency or data constraints. Sub-100 ms time-to-first-token, or data that cannot leave your VPC. Neither is a cost argument, and pretending otherwise muddies the decision.
How should you model the transition point?
Compute cost per million tokens at four utilisation levels, then find where it drops below your hosted API rate. That utilisation level is your threshold. If current traffic gives you 30% and you need 55%, you are one growth phase away — build the capability, keep paying for the API today, and migrate when the logs show you crossed it.
Do not migrate in anticipation. Provisioning a GPU for traffic you hope to have is how a $400/month bill becomes $1,200.
Key Takeaways
- Cost per request = (GPU hourly / utilisation) × seconds per request, divided by 3600.
- Utilisation is the dominant variable; 10% utilisation costs 6x a hosted API for the same output.
- An 8B model on an L4 at 60% utilisation lands near $0.32 per million output tokens.
- Add 0.2-0.5 FTE for serving engineering — decisive at low volume, noise at high volume.
- Self-hosting wins on high sustained volume, fine-tuned small models, or latency/data constraints. Not on “GPUs are cheap”.
