How to Run Batch GPU Inference on Modal Without a Server

Batch inference — running a model over thousands of rows, images or documents — is a classic case for Modal. You fan the work out across many containers, each with a GPU if needed, and pay only for the seconds each one runs. No server sits idle waiting for the next batch.

Start with the basic Modal deployment guide if you have not run a function yet. This article covers the scaling and cost patterns specific to batch jobs.

How do you parallelize a batch in Modal?

Use a function that takes one item and a driver that maps it over the whole list. Modal lets a function spawn many parallel containers, so the wall-clock time depends on concurrency, not list length.

1. Write a process_one(item) function with the GPU decorator.
2. In a driver function, call process_one.map(items).
3. Set max_containers to control how wide it scales.
4. Collect results as they return.

A batch of 10,000 short items can finish in minutes if you let it scale wide.

How do you avoid loading the model repeatedly?

Loading a large model on every call wastes time and money. Modal keeps containers warm while a batch runs and can cache the model weights in a volume or the image cache. Load the model once at container start and reuse it for every item that container handles.

How do you keep costs predictable?

• Cap concurrency with max_containers so a big list cannot spin up hundreds of GPUs.
• Use gpu="any" to accept cheaper hardware when the model tolerates it.
• Right-size the GPU — a small model on an A100 is money burned.
• Test on 100 items before running 100,000.

How do you store and read results?

Write outputs to a Modal volume for persistence, or push to your own S3/GCS bucket. Volumes survive between runs, so a crashed batch can resume instead of restarting. For huge datasets, stream results out rather than holding everything in memory.

Modal vs. a rented GPU box

If your workload runs continuously, a rented GPU instance is cheaper and simpler. Modal wins when the job is intermittent — nightly embeddings, a weekly re-index, occasional fine-tuning. You skip provisioning, and the meter stops when the batch ends. The trade-off is cold-start latency and a platform to learn.

Key Takeaways

• Modal maps one function over a list to run batch inference in parallel.
• Cache model weights so each container loads them once.
• Cap concurrency with max_containers to control cost.
• Use volumes to persist results and resume failed batches.
• Best for intermittent jobs; a continuous workload is cheaper on a rented box.

Last updated October 2026. More at Modal.