How to Add Fallback and Load Balancing with the Portkey Gateway
Portkey is an AI gateway: your app calls it instead of calling OpenAI, Anthropic or Gemini directly, and it decides which provider serves the request, retries on failure, and falls back when one is down.
The migration is one line in most SDKs — change base_url. Everything else is configuration on the gateway side, which means you can change routing behaviour without shipping code.
What does an AI gateway do that a plain SDK does not?
Four things that matter the day a provider has an outage:
-
Fallback. Primary returns a 500 or times out, the gateway retries against a second provider or a second key.
-
Load balancing. Weighted routing across keys or providers, so one account’s rate limit does not cap your product.
-
Retries with backoff. Automatic retry on 429 and 5xx, with the request body preserved.
-
Observability. Every call logged with latency, tokens, cost and which provider actually served it.
Who this is for: anything in production where the model call can fail, and any team using two providers for cost or capability reasons. Who it is not for: local prototypes, or workloads where adding a network hop in the request path is unacceptable.
How do you point your app at the gateway?
Create a gateway config in the Portkey dashboard, then set the base URL:
from openai import OpenAI
client = OpenAI(
base_url="https://api.portkey.ai/v1",
api_key="PORTKEY_KEY",
default_headers={
"x-portkey-provider": "@openai-prod",
"x-portkey-virtual-key": "vk-...",
},
)
The virtual-key holds the real provider key on Portkey’s side, so your application never ships an OpenAI key. That alone is a reason to use a gateway: rotating a leaked key becomes a dashboard edit instead of a redeploy across services.
How do you configure fallbacks?
Fallback chains are declared in a config object with a strategy of fallback:
{
"strategy": { "mode": "fallback" },
"targets": [
{ "virtual_key": "openai-prod", "override_params": { "model": "gpt-4o" } },
{ "virtual_key": "anthropic-prod", "override_params": { "model": "claude-sonnet-4" } }
]
}
The gateway tries the first target; on a retryable error it moves to the second. Pass this config via the config header or as a saved config ID.
Three settings decide whether failover helps or hurts:
-
Retry count. Two retries per target is a sane default. Five retries means a user waits 30 seconds before seeing an error.
-
Timeout per attempt. Set a hard timeout (10–30 seconds depending on your task). Without it, a hung provider consumes the whole budget before the fallback fires.
-
Which errors are retryable. 429 and 5xx: yes. 400 (bad request): no, and retrying a malformed request on a second provider just doubles your error logs.
How does weighted load balancing work in practice?
Use strategy.mode: "loadbalance" with weights. Two legitimate uses:
-
Multiple keys for the same provider to raise your effective rate limit. Split 50/50 across two orgs.
-
Cost splitting across providers — route 70% to a cheaper model and 30% to a stronger one, measured by an eval score you trust.
The trap with option two: the two providers produce different outputs for the same prompt, and your users will notice inconsistent tone before you notice a metric move. If you use load balancing across providers, test both paths with the same eval set and only mix where quality is genuinely equivalent.
Sticky sessions matter for multi-turn chat. Splitting turns across providers mid-conversation loses prompt-caching benefits and can shift the reply style mid-thread. Route by session hash where the SDK supports it.
What does the extra hop cost in latency?
Portkey’s gateway adds a small overhead per request — in the tens of milliseconds range for the routing decision — plus whatever the provider takes. That is usually noise compared to a 1–3 second generation.
Where it is not noise: streaming. The first token now travels through an additional hop, and if the gateway buffers the stream, your time-to-first-token gets worse. Verify streaming passes through unbuffered before you ship.
How do you tune caching on top of this?
A gateway is also the right place to put a response cache. Exact-match caching on repeated prompts is safe and cheap; semantic caching is riskier, because two questions with similar embeddings can need different answers, especially with any personalisation.
For a document Q&A product, exact-match caching on the (prompt, model, temperature) tuple typically hits 10–30% of traffic from retries and refreshes alone. Do not cache streaming responses unless the gateway can replay them without breaking the client.
What should you check before production?
Run a fault injection test. Point the primary at an invalid key and confirm the fallback serves the request; then add artificial latency to the primary and confirm your timeout fires before the user’s patience does. Teams frequently configure fallbacks and never test them, then discover during an outage that both targets share the same virtual key.
Quick Summary
-
Portkey is an OpenAI-compatible gateway: changing
base_urland adding a virtual key gets you retries, fallbacks and logs. -
Virtual keys keep provider API keys out of your application entirely, which makes rotation a dashboard edit.
-
Use two retries and a 10–30 second per-attempt timeout; unlimited retries just delay the error the user sees.
-
Only load-balance across providers where an eval shows equivalent quality, and keep chat sessions sticky.
-
Test the fallback path with a deliberately broken primary key before relying on it.
