Routing and reliability
Routing and failover
How one model slug reaches several providers, and what happens when one is slow, throttled or down.
A model slug points at a waterfall: an ordered list of routes (rungs) that can serve that model. The gateway sends your request to the first rung that can take it and moves down the list on provider failures, before any output is sent. You get one response in the API shape you called, and headers that name the rung that answered.
The waterfall
Each rung is one way to reach the model. The kind of rung decides who pays.
| Rung | Paid by | Set up by |
|---|---|---|
| Platform-funded | Your credits, at catalog price | Experiential |
| Your own key (BYOK) | Your provider, directly | You, see Bring your own key |
| Your own endpoint | You | You, a private model in your org |
Public models keep the platform order until your organization connects its own key. Then you can reorder the chain on the model page, or through the waterfall API.
| At a glance | |
|---|---|
| Unit | One model slug |
| Order | Top rung first |
| Failover | Before the first output byte only |
| Markup | None on either lane |
What happens on a failure
Failover only happens before a provider has committed to the request. Once output starts, the answer comes from that rung, and the gateway never splices two providers into one response.
| Upstream answer | What the gateway does |
|---|---|
| Transport error, timeout, 5xx before output | Retries the rung once, then moves to the next rung |
| 429 (provider throttle) | Moves on, waits for a short same-rung redial, or returns the throttle, per the model's policy |
| Authentication failure on your key | Drops that rung and uses the next rung that is on |
| 400 invalid request | Returns it to you. A bad request is not retried elsewhere |
| Policy refusal | Returns refusal. The same content is refused again |
| Every rung fails | 502 all_routes_failed, see Errors |
HTTP/2 200
x-request-id: req_01HZX4...
x-gateway-provider: anthropic
x-gateway-route-depth: 1
x-gateway-route-reason: direct
x-gateway-zdr: falsex-gateway-route-depth: 1 means the second rung served. These headers are set only after a provider commits, so they never describe a rung that failed first.
Slow lanes and unhealthy keys
The gateway watches every rung's live behavior and adjusts the order on its own. You do not reorder chains by hand for speed.
| Condition | Effect | Recovery |
|---|---|---|
| A rung is much slower to first token, or decodes slowly | Moved behind its healthy siblings for new requests | Returns to its place automatically once it is fast again |
| A rung keeps answering 429 | Sits out briefly; only that rung, not the whole provider | Automatic, after a short window |
| Your key is rejected or out of quota | That key's rung is skipped | A successful key check or the cooldown ends |
| Your key is the only rung and cannot serve | The request fails and names the account | Fix the key or add another rung |
A demoted rung is never the last one: if it is the only way to reach the model, it still serves. A request that continues sealed reasoning from an earlier turn stays on the rung that issued it.
| Signal | Read it from |
|---|---|
| Which rung served | x-gateway-provider, body provider |
| How far down | x-gateway-route-depth |
| How many tries | attempt_count on the usage event |
/api/gateway/usage/eventsCache affinity
Prompt caching is cheaper and faster when a conversation keeps hitting the provider that holds its cache. So the gateway keeps a conversation on the same rung while that rung is healthy. When it is not, the gateway fails over and takes the cache miss: staying available beats a warm cache. See Caching for prices.
| Situation | Choice |
|---|---|
| Rung healthy, holds your cache | Stay |
| Rung down or throttled | Fail over, pay a cache miss |
| Rung slow | Demoted for new conversations |
Per-request controls
Add a gateway object to a Chat Completions, Responses or Messages body to bound retries for one call. It is never sent upstream. Without it, a request gets up to two attempts per rung and eight in total.
| Field | Range | Meaning |
|---|---|---|
retry.max_attempts_per_route | 1 to 4 | Attempts on one rung, first try included |
retry.max_total_attempts | 1 to 8 | Attempts across all rungs |
retry.backoff | none or exponential | Waiting between same-rung attempts |
routing.allow_fallbacks | boolean | false keeps the call on one rung |
routing.route_id | opaque handle | Prefer, or pin, one rung |
Get route_id values from the model's providers listing with your key. These controls never bypass credit, rate, privacy or region rules. Set max_retries=0 in your SDK if the gateway should own retries, or the two layers multiply.
/api/models/{slug}/providers{
"model": "claude-sonnet-5.5",
"messages": [{ "role": "user", "content": "Hello" }],
"gateway": {
"retry": {
"max_attempts_per_route": 1,
"max_total_attempts": 4,
"backoff": { "type": "none" }
},
"routing": { "allow_fallbacks": true }
}
}client.chat.completions.create(
model="claude-sonnet-5.5",
messages=[{"role": "user", "content": "Hello"}],
extra_body={"gateway": {"retry": {"max_total_attempts": 4}}},
)Related
- Choosing a model for picking slugs.
- Rate limits for 429s the gateway itself returns.
- Data controls: a zero data retention policy removes rungs before routing.