Add model fallback

The pattern is older than LLMs: try the cheap path first, fall back to the expensive one when it breaks. With a single LLM you would wrap the call in a retry loop, catch the 503, swap the model id and call again, all while tracking which provider is currently flaky. Ringside folds that loop into the request. You pass an ordered list of models, the gateway walks it, and you get back the first success.

The point is cost. Put a small, fast model first (say fc:openai/gpt-4o-mini) and a stronger one behind it (fc:openai/gpt-4o). Most traffic is served by the cheap model. The strong model only earns its price when the cheap one is unavailable or produces output that fails a schema check. You pay for capability only on the requests that actually need it.

The mental model

A normal chat call sets model to one prefixed ref. To enable a cascade you set model to an array of refs. Each entry follows the same grammar as a single ref: it must carry an fc:, match:, slot: or dyn: prefix. A bare name like gpt-4o is rejected with 400 invalid_model_ref, in an array exactly as it is for a string.

{ "model": "fc:openai/gpt-4o-mini", "messages": [ ... ] }

{ "model": ["fc:openai/gpt-4o-mini", "fc:openai/gpt-4o"], "messages": [ ... ] }

Ringside calls the first entry. On success it returns immediately. On a failure that is worth retrying it moves to the next entry and tries again. It keeps going until a model succeeds or the chain is exhausted. The whole thing is one HTTP request from your side; the internal attempts are invisible except through the response headers and your usage log.

  POST /v1/chat/completions
  model: ["fc:openai/gpt-4o-mini", "fc:openai/gpt-4o"]
        |
        v
  +-----------------------------+
  | try fc:openai/gpt-4o-mini   |--- 503 upstream_unavailable ---+
  +-----------------------------+                                |
        escalate <-------------------------------------------------+
        |
        v
  +-----------------------------+
  | try fc:openai/gpt-4o        |--- 200 OK --> return
  +-----------------------------+
        |
        v
  X-Ringside-Model-Used: fc:openai/gpt-4o
  X-Ringside-Models-Tried: fc:openai/gpt-4o-mini,fc:openai/gpt-4o
  X-Ringside-Fallback-Triggered: true

When the gateway escalates

The distinction that makes fallback safe is which failures escalate and which stop the chain. A provider outage should move to the next model. A malformed request should not, because the next model would fail the same way and you would just burn the whole chain on one bad call.

Escalates to the next model

Stops the chain (returned as-is)

Reading the result

On a cascade Ringside sets these headers so you can attribute the response and watch fallback behaviour in production. X-Ringside-Models-Tried and X-Ringside-Fallback-Triggered appear on both the success path and on a non-escalating failure that ends the chain. X-Ringside-Model-Used is set on success only, since a failure has no winning model.

X-Ringside-Model-Used

The model that actually served the response. Only set on success. This is the ref to log for cost attribution and for spotting a model that fails constantly.

X-Ringside-Models-Tried

Comma-separated list of every model attempted, in order, e.g. fc:openai/gpt-4o-mini,fc:openai/gpt-4o. A single entry means the first model served.

X-Ringside-Fallback-Triggered

true when escalation occurred (the served model was not the first in the chain), false when the first model served. Track the rate of true as your cheap-model health signal.

There is a fourth header you only see when the entire chain fails: X-Ringside-Fallback-Exhausted: true, set alongside X-Ringside-Models-Tried on the last attempt's error response. The body of that response is the last model's error, so a fully exhausted cheap+strong pair surfaces the strong model's 503.

Every attempt also writes its own usage record (a failed attempt and the eventual success both land in your log), so the dashboard at /app/logs shows precisely which models were tried on a given request id and why each one bailed. The request id itself is the X-Request-Id response header, never a body field.

Worked example: curl

Cheap first, strong behind it, attributed to a customer. Note the -i flag so you see the response headers that tell you which model won.

curl -i https://api.fightclub.pro/v1/chat/completions \
  -H "Authorization: Bearer $FC_API_KEY" \
  -H "FC-Customer: cus_42" \
  -H "Content-Type: application/json" \
  -d '{
    "model": ["fc:openai/gpt-4o-mini", "fc:openai/gpt-4o"],
    "messages": [{"role": "user", "content": "Summarise this ticket in one line."}],
    "max_tokens": 64
  }'

When the cheap model is healthy you get a 200 and these headers, and the strong model is never called or billed:

HTTP/1.1 200 OK
X-Request-Id: req_a1b2c3d4
X-Ringside-Model-Used: fc:openai/gpt-4o-mini
X-Ringside-Models-Tried: fc:openai/gpt-4o-mini
X-Ringside-Fallback-Triggered: false
X-Ringside-Wallet-Deducted: 0.000118
X-Ringside-Wallet-Balance: 41.882300

When OpenAI is throwing 5xx for the mini model, the same request escalates and the strong model serves it. You did nothing different; the headers tell the story:

HTTP/1.1 200 OK
X-Request-Id: req_e5f6a7b8
X-Ringside-Model-Used: fc:openai/gpt-4o
X-Ringside-Models-Tried: fc:openai/gpt-4o-mini,fc:openai/gpt-4o
X-Ringside-Fallback-Triggered: true

Worked example: SDK

The OpenAI SDK is a drop-in: point base_url at the Ringside API and pass the array straight through as model. The SDK does not type model as a list, so cast or ignore the type hint; the wire payload is what matters. Read the fallback headers off the raw response.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.fightclub.pro/v1",
    api_key=os.environ["FC_API_KEY"],
)

# .with_raw_response gives access to the response headers
raw = client.chat.completions.with_raw_response.create(
    model=["fc:openai/gpt-4o-mini", "fc:openai/gpt-4o"],  # array = cascade
    messages=[{"role": "user", "content": "Summarise this ticket in one line."}],
    max_tokens=64,
    extra_headers={"FC-Customer": "cus_42"},
)

print("served by:", raw.headers["X-Ringside-Model-Used"])
print("tried:    ", raw.headers["X-Ringside-Models-Tried"])
print("escalated:", raw.headers["X-Ringside-Fallback-Triggered"])

completion = raw.parse()
print(completion.choices[0].message.content)

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://api.fightclub.pro/v1',
  apiKey: process.env.FC_API_KEY,
});

const { data, response } = await client.chat.completions
  .create(
    {
      // cast through any: the chain is valid on the wire, the SDK types model as a string
      model: ['fc:openai/gpt-4o-mini', 'fc:openai/gpt-4o'] as any,
      messages: [{ role: 'user', content: 'Summarise this ticket in one line.' }],
      max_tokens: 64,
    },
    { headers: { 'FC-Customer': 'cus_42' } },
  )
  .withResponse();

console.log('served by:', response.headers.get('X-Ringside-Model-Used'));
console.log('escalated:', response.headers.get('X-Ringside-Fallback-Triggered'));
console.log(data.choices[0].message.content);

Attribution and metering

Fallback does not change how spend is metered. Each attempt that reaches a provider is metered like any chat call: against your developer wallet, or against the customer wallet when FC-Customer is present. A failed attempt that errored at the provider still records a usage event, so a chain that escalates can show two rows in your log for one request id. The headers X-Ringside-Wallet-Deducted and X-Ringside-Wallet-Balance on the final response report the charge and remaining balance for the winning call.

Attribute the call the usual way. Set FC-Customer to a cus_ id, an ext:<external_id> or a plain external id string, and add FC-Customer-Strict: true if you want unknown customers rejected instead of lazily created. If a chain hits the customer budget ceiling it returns 402 customer_budget_exceeded and stops; it does not try to find a model the customer can afford.

Constraints and failure modes

Two combinations are rejected at parse time, before any model is called, because the cascade semantics are unsound with them.

There is a third interaction worth knowing. A response_format of { type: "json_schema" } combined with stream: true returns 400 streaming_with_schema_fallback_unsupported, independent of the chain. But a schema with a non-streaming cascade is the intended use: a model whose output fails the schema after its re-prompts returns 422 response_schema_validation_failed, which is an escalation trigger, so the next model gets a shot at producing valid JSON.

Limits and defaults

Chain length

The array must hold at least one ref (an empty array is treated as a missing model). Order is the escalation order; entry 0 is tried first.

Entry grammar

Every entry needs an fc: / match: / slot: / dyn: prefix and may carry an @<region> residency suffix on fc: entries only. A bare or bad ref is 400 invalid_model_ref.

stream default

false. A cascade requires the default (no streaming).

moderation default

none. Moderation, if you set it, is applied per attempt.

Messages size

The serialized messages array is capped at 100,000 bytes, same as a single-model chat call.

When not to use it

• ·You need streaming. Stream a single model; you cannot stream a cascade.
• ·You need idempotency on the call. Pick one model and pass the idempotency-key header; the cascade and the key are mutually exclusive.
• ·You want failover across data residency regions for the same logical model. That is a match: resolution and routing concern, not a quality cascade. A chain is about trying a different (usually stronger) model, not the same model in another region.
• ·Your first model is failing on 4xx, not 5xx. A 400/401/402/404 stops the chain by design, so a cascade does nothing for a request that is itself malformed or unfunded.

Related