How to Deploy an Open Model to Production with Baseten

Serving your own model sounds cheap until you price the idle GPU. Baseten’s pitch is that it will scale to zero and back without the 8-minute cold start that makes self-hosting impractical for bursty traffic.

Direct answer: to deploy a model on Baseten, package it as a Truss (a folder with config.yaml and a model/model.py defining predict), run truss push to build and deploy it, then call the returned HTTPS endpoint. Billing is per second of inference on the Pro plan, with a free trial credit of about $30.

What does the Truss structure look like?

Four files. config.yaml declares the environment, resources, and model weights. model/model.py holds a class with a predict method. requirements.txt lists Python dependencies. packages/ is optional and holds any vendored code.

# config.yaml
model_name: llama-3-1-8b
resources:
  accelerator: L4
  cpu: "4"
  memory: 16Gi
model_metadata:
  custom: {}

L4 is the cheapest GPU option and fits an 8B model in 4-bit. A10G costs about 2.5x and fits 13B. H100 is for 70B and above, and the price step between A10G and H100 is large enough that you should be confident the quality gap justifies it.

How does the predict function work?

Baseten wraps a class. Load the model in __init__ so it happens once per container, not once per request.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

class Model:
    def __init__(self):
        self.tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
        self.model = AutoModelForCausalLM.from_pretrained(
            "meta-llama/Llama-3.1-8B-Instruct",
            torch_dtype="bfloat16", device_map="cuda"
        )

    def predict(self, model_input):
        prompt = model_input.pop("prompt")
        inputs = self.tok(prompt, return_tensors="pt").to("cuda")
        out = self.model.generate(**inputs, max_new_tokens=model_input.get("max_tokens", 256))
        return {"text": self.tok.decode(out[0], skip_special_tokens=True)}

Downloading weights in __init__ reads from the Baseten model cache, which persists between cold starts. First build pulls from Hugging Face and takes minutes; subsequent starts are close to instant.

How do you build and deploy?

pip install truss
truss login
truss push --publish

--publish makes the endpoint production and gives you the stable URL. Without it you get a development deployment that sleeps after inactivity and wakes on the next request.

Development deployments are the right place to iterate — cheap, and the wake latency does not matter while you are testing. Promote to production once the model answers correctly.

Builds take 5-15 minutes depending on image size. If your requirements.txt pulls torch from PyPI (about 2 GB), expect the long end. Baseten’s base images already include CUDA and torch, so leave them out to halve build time.

What does autoscaling actually do here?

Scale-to-zero means an unused deployment costs nothing beyond storage. The trade-off is cold start: a container that has been asleep needs to restart the process and reload weights, which for an 8B model on cached weights lands around 30-60 seconds. Baseten keeps a warm pool option that removes this at extra cost.

Concurrency matters more. An L4 serving an 8B model handles roughly 8-12 concurrent requests before latency doubles. Set max_concurrency above that and you get queueing that looks like a hang.

For a product with steady traffic, keep one replica warm and skip the cold start entirely. For a demo or an internal tool used twice a day, scale-to-zero is exactly right.

Is it cheaper than a hosted API?

Do the arithmetic on your own numbers, but here is the shape of it. An L4 at Baseten’s Pro rate runs about $0.6-0.8 per GPU-hour. At 200 requests per hour with 500 output tokens each, that is roughly $0.004 per request in compute — assuming the GPU is actually busy.

Idle capacity is the killer. At 20 requests per hour, the same GPU still costs the same per hour while serving a tenth of the work, and the per-request cost jumps to $0.04. Hosted APIs win below roughly 50% GPU utilisation.

Self-hosting wins when: you fine-tuned a model nobody hosts, you need sub-200 ms time-to-first-token on a specific GPU, or your volume is high enough to keep the card saturated. Otherwise the API is simpler and often cheaper.

Where it falls short

You own the correctness of the serving stack. If the model outputs garbage at high concurrency, that is your bug, not theirs. There is no equivalent of an API provider’s versioning guarantee — pin your model weights explicitly or an upstream Hub update changes your behaviour silently.

Logging is also thinner than a full observability platform. You get request logs and metrics; you bring the tracing. For most teams that is fine. For regulated environments, check the data-residency story before committing.

Key Takeaways

• A Truss is four files; model/model.py with a predict method is the whole contract.
• Load weights in __init__ and rely on the model cache — it turns a 5-minute cold start into 30 seconds.
• L4 fits 8B at 4-bit, A10G fits 13B, H100 is for 70B and the price jump is steep.
• Self-hosting beats hosted APIs only above roughly 50% GPU utilisation.
• Keep one warm replica for steady traffic; scale to zero only for sporadic use.