Routing and reliability

Routing and failover

How one model slug reaches several providers, and what happens when one is slow, throttled or down.

A model slug points at a waterfall: an ordered list of routes (rungs) that can serve that model. The gateway sends your request to the first rung that can take it and moves down the list on provider failures, before any output is sent. You get one response in the API shape you called, and headers that name the rung that answered.

The waterfall

Each rung is one way to reach the model. The kind of rung decides who pays.

RungPaid bySet up by
Platform-fundedYour credits, at catalog priceExperiential
Your own key (BYOK)Your provider, directlyYou, see Bring your own key
Your own endpointYouYou, a private model in your org

Public models keep the platform order until your organization connects its own key. Then you can reorder the chain on the model page, or through the waterfall API.

At a glance
UnitOne model slug
OrderTop rung first
FailoverBefore the first output byte only
MarkupNone on either lane

What happens on a failure

Failover only happens before a provider has committed to the request. Once output starts, the answer comes from that rung, and the gateway never splices two providers into one response.

Upstream answerWhat the gateway does
Transport error, timeout, 5xx before outputRetries the rung once, then moves to the next rung
429 (provider throttle)Moves on, waits for a short same-rung redial, or returns the throttle, per the model's policy
Authentication failure on your keyDrops that rung and uses the next rung that is on
400 invalid requestReturns it to you. A bad request is not retried elsewhere
Policy refusalReturns refusal. The same content is refused again
Every rung fails502 all_routes_failed, see Errors
HTTP/2 200
x-request-id: req_01HZX4...
x-gateway-provider: anthropic
x-gateway-route-depth: 1
x-gateway-route-reason: direct
x-gateway-zdr: false

x-gateway-route-depth: 1 means the second rung served. These headers are set only after a provider commits, so they never describe a rung that failed first.

Slow lanes and unhealthy keys

The gateway watches every rung's live behavior and adjusts the order on its own. You do not reorder chains by hand for speed.

ConditionEffectRecovery
A rung is much slower to first token, or decodes slowlyMoved behind its healthy siblings for new requestsReturns to its place automatically once it is fast again
A rung keeps answering 429Sits out briefly; only that rung, not the whole providerAutomatic, after a short window
Your key is rejected or out of quotaThat key's rung is skippedA successful key check or the cooldown ends
Your key is the only rung and cannot serveThe request fails and names the accountFix the key or add another rung

A demoted rung is never the last one: if it is the only way to reach the model, it still serves. A request that continues sealed reasoning from an earlier turn stays on the rung that issued it.

SignalRead it from
Which rung servedx-gateway-provider, body provider
How far downx-gateway-route-depth
How many triesattempt_count on the usage event
GET/api/gateway/usage/events

Cache affinity

Prompt caching is cheaper and faster when a conversation keeps hitting the provider that holds its cache. So the gateway keeps a conversation on the same rung while that rung is healthy. When it is not, the gateway fails over and takes the cache miss: staying available beats a warm cache. See Caching for prices.

SituationChoice
Rung healthy, holds your cacheStay
Rung down or throttledFail over, pay a cache miss
Rung slowDemoted for new conversations

Per-request controls

Add a gateway object to a Chat Completions, Responses or Messages body to bound retries for one call. It is never sent upstream. Without it, a request gets up to two attempts per rung and eight in total.

FieldRangeMeaning
retry.max_attempts_per_route1 to 4Attempts on one rung, first try included
retry.max_total_attempts1 to 8Attempts across all rungs
retry.backoffnone or exponentialWaiting between same-rung attempts
routing.allow_fallbacksbooleanfalse keeps the call on one rung
routing.route_idopaque handlePrefer, or pin, one rung

Get route_id values from the model's providers listing with your key. These controls never bypass credit, rate, privacy or region rules. Set max_retries=0 in your SDK if the gateway should own retries, or the two layers multiply.

GET/api/models/{slug}/providers
{
  "model": "claude-sonnet-5.5",
  "messages": [{ "role": "user", "content": "Hello" }],
  "gateway": {
    "retry": {
      "max_attempts_per_route": 1,
      "max_total_attempts": 4,
      "backoff": { "type": "none" }
    },
    "routing": { "allow_fallbacks": true }
  }
}
client.chat.completions.create(
    model="claude-sonnet-5.5",
    messages=[{"role": "user", "content": "Hello"}],
    extra_body={"gateway": {"retry": {"max_total_attempts": 4}}},
)