LLM API Fallback Chains: Survive Any Provider Outage

TL;DR — LLM provider outages are infrastructure facts, not surprises: on 3 September 2026 OpenAI, Anthropic, and Google all disrupted services within hours, and Claude's API had ten incidents across twelve days in June 2026. A fallback chain — SDK retries, then a cheaper model on the same provider, then a different provider — keeps your app answering when one provider can't. This guide shows all three layers with working Python code.

On 3 September 2026, three of the biggest AI providers stumbled inside the same few hours. OpenAI detected elevated error rates across ChatGPT and Codex around 10:58 UTC and needed about 24 minutes to apply mitigations; Anthropic's Claude Sonnet 5 had elevated errors the day before, resolved in roughly 25 minutes; Google's Gemini took around 2 hours to fully recover. That June, Claude's API and Claude Code went through a major incident on the 2nd, with status history recording more disruptions through the 16th — ten incidents in twelve days. OpenAI's own status history shows separate incidents on July 22, 23, 24, and 25.

None of this means any provider is uniquely unreliable. It means model APIs are infrastructure, and every infrastructure dependency eventually returns errors, slows down, or becomes unreachable. Independent monitoring puts major AI providers at 99.9% availability across 2026 — which sounds airtight until you multiply: 0.1% of 8,760 hours is about 8.8 hours of downtime a year. The question is never "will my provider go down?" but "what does my app do when it does?"

The three layers of a fallback chain#

A fallback chain is an ordered list of attempts, not a single retry. Think of it as three rungs:

1. Same request, same provider. The SDK retries transient errors — connection drops, timeouts, 429 rate limits, 5xx server errors. Free and automatic, but it only helps with seconds-long blips.
2. Model-tier failover. The provider is reachable but your model is not — rate-limited, capacity-constrained, or degraded. You switch to a cheaper or smaller model on the same provider, with the same key and billing.
3. Cross-provider failover. The whole provider is down. You send the request to a different provider's model entirely, with a different key.

Most production outages that hurt are rung 2 and 3, and most apps only have rung 1. The code below adds the other two.

Layer 1: let the SDK do its job#

The official OpenAI Python SDK already retries connection errors, 408 timeouts, 409 conflicts, 429 rate limits, and 5xx errors — 2 times by default, with short exponential backoff. You can tune it per client or per request:

from openai import OpenAI

client = OpenAI(
    api_key="sk-...",
    max_retries=3,  # default is 2; 0 disables retries
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Extract the invoice total."}],
)

Two gotchas. First, a retry that keeps hammering a failing endpoint can become its own outage amplifier — keep the retry budget small and move to failover instead of increasing it past 3. Second, not every 429 is worth retrying: a quota-exhausted 429 never recovers by waiting. When the error body distinguishes rate limits from quota or billing faults, route on that distinction instead of retrying blindly.

Layer 2 and 3: a hand-rolled fallback loop#

When retries are exhausted, walk the chain. This pattern works with any OpenAI-compatible client, including tokens.bd — you only change the base_url and the key:

import openai, time

def chat_with_fallback(client, chain, messages, timeout=20.0):
    """chain: list of (model_name) tried in order; returns (model, response)."""
    last_error = None
    for model in chain:
        try:
            response = client.with_options(
                max_retries=0,          # retries handled by the chain, not the SDK
                timeout=timeout,
            ).chat.completions.create(model=model, messages=messages)
            return model, response
        except (openai.RateLimitError, openai.InternalServerError,
                openai.APITimeoutError, openai.APIConnectionError) as e:
            last_error = e              # retriable: try the next rung
            time.sleep(1)
        except openai.APIStatusError:
            raise                       # 400/401/403/404 won't heal — fail fast
    raise last_error

model, resp = chat_with_fallback(
    client,
    chain=["claude-sonnet-4-5", "claude-3-5-haiku", "gpt-4o"],
    messages=[{"role": "user", "content": "Summarize this diff."}],
)
print(f"Served by {model}:", resp.choices[0].message.content[:200])

The except openai.APIStatusError: raise line matters: 400/401/403/404 errors are deterministic — no fallback will fix a bad key or a malformed request, so fail loudly instead of burning a second provider's quota on a request that can never succeed. Log which model actually served each request; when you review an incident later, "served by" is the column that tells you whether the chain did its job.

The shortcut: OpenRouter's model array#

If you route through OpenRouter, the chain moves server-side. Send a models array instead of a single model — OpenRouter tries the primary, then falls through the array in order on downtime, rate limits, moderation refusals, or validation errors, and the response's model field tells you which one answered:

response = client.chat.completions.create(
    model="anthropic/claude-sonnet-4-5",
    extra_body={
        "models": [
            "anthropic/claude-sonnet-4-5",
            "openai/gpt-4o",
            "google/gemini-2.5-flash",
        ],
    },
    messages=[{"role": "user", "content": "Extract the invoice total."}],
)
print("Served by:", response.model)

Two details worth knowing. The fallback list is only consulted when the primary actually fails — you are not paying for standby. And OpenRouter's provider routing has an allow_fallbacks flag (on by default) that controls whether a model's request may move between providers behind the same slug; set it to false when you need a request to stay on one provider for compliance reasons.

The self-hosted option: LiteLLM Router#

If you prefer to own the chain, LiteLLM's Router maps logical model names to deployments and defines fallbacks per name. Fallbacks are tried in order, and the first succeeding model's response is returned:

from litellm import Router

model_list = [
    {"model_name": "primary",
     "litellm_params": {"model": "openai/gpt-4o", "api_key": "sk-openai"}},
    {"model_name": "backup",
     "litellm_params": {"model": "anthropic/claude-3-5-haiku-20241022",
                        "api_key": "sk-ant"}},
]

router = Router(
    model_list=model_list,
    fallbacks=[{"primary": ["backup"]}],  # rung 3: different provider
    num_retries=2,
    cooldown_time=60,                      # cool down a failing deployment
)

response = router.completion(
    model="primary",
    messages=[{"role": "user", "content": "Hello"}],
)

LiteLLM also supports context_window_fallbacks (used when a request exceeds the primary's context window) and content_policy_fallbacks (used on moderation refusals) — failover conditions you can't express with a plain retry loop.

Five rules that keep fallbacks from backfiring#

1. Set a per-rung timeout, not just a per-request one. Three rungs at 60 seconds each is a 3-minute response your user already closed. Short timeouts on the first rungs keep total latency bounded.
2. Log which model served every request. A silent fallback hides degradation: if your "GPT" endpoint is quietly answering with a cheaper model for a week, you want to know from the logs, not from a customer.
3. Never retry deterministic errors. 400, 401, 403, 404 — and quota-exhausted 429s — will not heal. Fail fast, surface them, fix the cause.
4. Add a cooldown, not just a queue. LiteLLM's cooldown_time stops hammering a failing deployment for a minute. A hand-rolled loop should do the same: after N consecutive failures, skip that rung for a while.
5. Test the chain by breaking it. Set max_retries=0, point the primary at an invalid key, and confirm the backup answers. A fallback you have never triggered is a fallback you do not have.

Where tokens.bd fits#

tokens.bd ships with automatic failover built in: requests are failed over to another upstream on 429, 5xx, and connection errors — rung 2 and part of rung 3 happen without any client code. That covers the provider-side failures in this post's evidence section; the client-side chain above still matters for the cases the gateway can't see, like a quota-exhausted key or a context-window overflow.

Two companion features make the chain safer in practice. Per-key monthly spend caps mean a runaway fallback loop — one that keeps succeeding expensively on the backup tier — gets cut off with a 403 monthly_spend_cap_exceeded instead of a surprise bill, and usage alerts at 50%, 75%, 90%, and 100% tell you a failover is running hot before the cap trips. If you run the fallback loop from the earlier section against tokens.bd, the only change is the client setup:

from openai import OpenAI

client = OpenAI(
    base_url="https://tokens.bd/v1",
    api_key="tk-...",
)
# chain: claude-sonnet-4-5 -> claude-3-5-haiku -> gpt-4o

The full model catalog, the API docs, and the gateway comparison cover setup, pricing in USD and BDT, and how tokens.bd's failover compares with OpenRouter's routing. For what a gateway actually is and when one replaces hand-rolled chains, see What Is an AI Gateway?.

FAQ#

Do I need fallbacks if my provider has 99.9% uptime? Yes — 99.9% is about 8.8 hours of downtime per year, and the failure doesn't have to be yours: the September 2026 simultaneous disruptions hit OpenAI, Anthropic, and Google within hours, partly through shared infrastructure. A fallback chain is the difference between your app riding through that and your users filing tickets.

What errors should trigger a failover? Timeouts, connection errors, 429 rate limits, and 5xx server errors. Don't fail over on 400, 401, 403, or 404 — those are deterministic and will fail on every provider. Quota-exhausted 429s also never recover; fail fast and surface them.

Isn't the OpenAI SDK's built-in retry enough? It covers seconds-long blips: 2 retries with exponential backoff on 408, 409, 429, 5xx, and connection errors. It can't switch models or providers, so it does nothing when your model is capacity-constrained or the whole provider is down. That's what rungs 2 and 3 are for.

Does OpenRouter's models array cost extra? No — the fallback list is only consulted when the primary fails, so you pay for the model that actually served the request. The response's model field tells you which one that was.

Should the fallback be a cheaper model? Usually, yes. A cheaper backup model on the same provider is the best value rung: it absorbs rate-limit and capacity events at a fraction of the cost. Reserve the cross-provider rung for full outages and billing lockouts, where any working model beats no model.

How do I test my fallback chain? Break the primary on purpose: set the SDK retries to 0, use an invalid key or a disabled model as rung 1, and confirm rung 2 answers. Check your logs for the "served by" line, and measure total latency to make sure per-rung timeouts keep it acceptable.

Outage data and SDK behavior checked 12 October 2026. Model slugs and API features change; the provider docs are the authority for current behavior, and the model catalog for models on Tokens.

Sources: Crypto Briefing — 3 Sept 2026 outages · SoftwareReviews — overlapping AI outages · fewshot-works/academy — June 2026 Claude outage analysis · openai-python README — retries · Microsoft Dev Proxy — testing OpenAI rate-limit handling · OpenRouter routing & fallbacks reference · LiteLLM routing integrations research · tokens.bd vs OpenRouter — automatic failover · tokens.bd docs