How to Set Up LiteLLM Proxy for 100+ LLM APIs

LiteLLM is an open-source library and proxy server that gives you one OpenAI-compatible endpoint for over 100 LLM APIs — OpenAI, Anthropic, Gemini, Mistral, Cohere and local models. Instead of rewriting code when you switch providers, you change a config line. The proxy mode also adds budgets, rate limits and logging across keys.

The Python package is free, and the self-hosted proxy is free. Only the hosted enterprise tier is paid. For most developers, LiteLLM is the layer that stops provider lock-in.

What is LiteLLM?

LiteLLM is a unified interface for calling LLMs. In library mode you import it and call completion() with a model string like anthropic/claude-3-5-sonnet. In proxy mode you run a server that speaks the OpenAI API and routes each request to the right provider.

How do you install the LiteLLM proxy?

1. pip install 'litellm[proxy]'
2. Create a config.yaml with your models and API keys.
3. Start it: litellm --config config.yaml --port 4000
4. Point your app at http://localhost:4000 instead of a provider URL.
5. Call it with any OpenAI client, setting the model to your configured alias.

Because the interface matches OpenAI, existing SDKs work without changes.

How do you define models and keys?

Each entry in config.yaml has a model name, the underlying provider model, and the credentials. You can set a fallback list so a failed model rolls over to another provider automatically. Virtual keys let you mint scoped tokens for teams, each with its own budget and rate limit.

What do budgets and logging give you?

The proxy tracks spend per key and can hard-stop a key when it hits its limit — useful when many services share one provider account. It logs every request and can push usage to tools like Langfuse, Helicone or Prometheus. That visibility is the main reason teams run the proxy instead of the bare library.

When should you use the library instead?

If you have a single app and no need for shared budgets, importing the library is simpler — no server to run. Use the proxy when multiple apps or teammates share providers, or when you want centralized logging and failover. Running the proxy for one script is overkill.

Key Takeaways

• LiteLLM gives one OpenAI-compatible endpoint for 100+ LLM providers.
• The library and self-hosted proxy are free; only enterprise hosting is paid.
• Install with pip install 'litellm[proxy]' and run on port 4000.
• Config supports fallbacks, virtual keys, budgets and rate limits.
• Use the library for single apps, the proxy for shared/multi-service setups.

Last updated October 2026. Specs at LiteLLM.