Idempotency and reliability

Why this exists

A chat completion is not free and it is not instant. It debits a wallet, meters usage and can take seconds against a slow upstream. So the classic at-least-once retry problem is real here: if your POST to /v1/chat/completions fails at the transport layer, retrying blindly risks paying for and storing two completions for one logical request.

The fix is an idempotency key. You attach a key you generated to the request. The first time Ringside serves a 2xx for that key, it stores the response body and status. Any later request with the same key inside the replay window short-circuits before the model runs and returns the stored response. The model executes once. The wallet is debited once. Your retry is safe.

The idempotency-key header

The header name is exactly idempotency-key, lowercase. You choose the value. A UUIDv4 per logical operation is the usual pattern. The key is opaque to Ringside, so any string your client treats as unique per operation works.

Scope matters for tenant safety. The stored record is keyed by (developer, customer, key), not just (developer, key). The customer dimension is the attribution target: the pinned customer id when the call is attributed (a Client token, or a Bearer call carrying an FC-Customer header) or the empty string for an unattributed developer-direct call. Without that dimension one customer could send a key another customer already used under the same developer and read back the other customer's completion. The customer scope closes that.

What a replayed response looks like

A replay returns the exact stored body with its stored status. It carries an X-Request-Id for the replay request itself, which differs from the original request's id because it is a fresh inbound request. The model is not called, no new usage event is recorded against the key claim and the wallet is not debited a second time.

The model fallback cascade

Idempotency makes a retry safe. The fallback cascade reduces how often you need one. Send model as an array instead of a string and Ringside walks the chain in order, calling the next model only when the current attempt fails for a reason worth escalating. Order the array cheap-to-expensive and you get a cheap primary that quietly upgrades to a stronger model when the cheap upstream is down or returns unusable output.

{
  "model": ["fc:groq/llama-3.1-8b", "fc:openai/gpt-4o-mini", "fc:anthropic/claude-sonnet-4"],
  "messages": [{ "role": "user", "content": "Summarize this contract clause." }],
  "response_format": { "type": "json_object" }
}

Each attempt is a real call against the underlying handler and records its own usage event, so your audit log shows precisely which models were tried and why each non-final attempt was skipped. A success returns immediately. The chain stops the moment an attempt returns something the cascade is not allowed to escalate past.

What triggers escalation

If every model in the chain fails, Ringside returns the last failure response and adds X-Ringside-Fallback-Exhausted: true alongside the chain headers, so an exhausted cascade is distinguishable from a single-model failure.

Headers that report the cascade

X-Ringside-Model-Used

The model ref that served the response, echoed exactly as you sent it in the chain (so an fc: ref comes back fc:-prefixed). Present on the success path.

X-Ringside-Models-Tried

Comma-separated list of every model attempted, in order. Present on success and on a non-escalating failure.

X-Ringside-Fallback-Triggered

true if the response came from a later model in the chain (escalation happened), false if the first model served it.

X-Ringside-Fallback-Exhausted

true only when the whole chain failed and the last error is being returned.

Mutual exclusivity

The cascade cannot combine with streaming or idempotency, and both rejections are deliberate rather than incidental.

Practically, you pick one tool per call. Use a cascade for a one-shot generation where automatic upgrade beats a manual retry. Use a single model plus an idempotency-key when you need exactly-once semantics on a retried operation. They solve different halves of the reliability problem and Ringside refuses to let them silently undermine each other.

A retry strategy

Build your retry logic around the error class, not the HTTP code alone, and always branch on the code field rather than the human-readable message. The rule is short. Retry transport failures and 5xx. Retry a 429 after waiting out its Retry-After. Never retry a 4xx validation error, because the same body will fail the same way every time.

Webhooks: at-least-once

Some outcomes you cannot poll for cheaply, like a batch finishing overnight or a customer crossing its monthly budget. Those arrive as webhooks. Webhook delivery is at-least-once with retries, which is the mirror image of request idempotency: Ringside guarantees it will keep trying, so your endpoint must tolerate seeing the same event more than once.

• ·Deduplicate on the event id in your handler. A redelivery of an event you already processed should be a no-op.
• ·Verify the X-FC-Signature header (t=<unix>,v1=<hex>) with a constant-time compare against the raw body, and reject stale timestamps.
• ·A sustained delivery-failure streak auto-disables the endpoint (status becomes disabled_after_failures); inspect GET /v1/webhooks/{id}/deliveries and re-enable once your receiver is healthy.
• ·Replay a specific missed delivery with POST /v1/webhooks/{id}/deliveries/{deliveryId}/retry.

The full signature scheme, the rotation grace window (two v1= digests during rotation) and the event taxonomy are covered on the webhooks reference. The request.failed_after_retry event in particular fires when Ringside's own internal retries for a request are exhausted, which is a useful signal to wire into alerting.

Worked example

A single attributed chat completion with a stable idempotency-key, retried safely. The key is generated once per operation and reused on every retry. The first attempt times out at your client; the second lands and either runs the model or replays the stored 200 if the first attempt had in fact reached Ringside.

curl https://api.fightclub.pro/v1/chat/completions \
  -H "Authorization: Bearer ko_<64-hex>" \
  -H "Content-Type: application/json" \
  -H "FC-Customer: cus_42" \
  -H "idempotency-key: 6f1c2a90-3b7e-4d1f-9a2c-7e51b0c4d8a1" \
  -d '{
    "model": "fc:openai/gpt-4o-mini",
    "messages": [{ "role": "user", "content": "Draft a one-line release note." }]
  }'

Now contrast the cascade form. Here the goal is automatic upgrade, not retry-safety, so there is no idempotency-key (the two cannot coexist) and stream is left at its default of false.

curl https://api.fightclub.pro/v1/chat/completions \
  -H "Authorization: Bearer ko_<64-hex>" \
  -H "Content-Type: application/json" \
  -H "FC-Customer: cus_42" \
  -d '{
    "model": ["fc:groq/llama-3.1-8b", "fc:openai/gpt-4o-mini", "fc:anthropic/claude-sonnet-4"],
    "messages": [{ "role": "user", "content": "Classify the sentiment." }]
  }'

# On a Groq outage the response carries, for example:
#   X-Ringside-Model-Used: fc:openai/gpt-4o-mini
#   X-Ringside-Models-Tried: fc:groq/llama-3.1-8b,fc:openai/gpt-4o-mini
#   X-Ringside-Fallback-Triggered: true

Defaults and limits

• ·Idempotency replay window: 24 hours from the first successful store.
• ·Idempotency scope: (developer, customer, key); the customer dimension is empty for unattributed developer-direct calls.
• ·Stored bodies cover 2xx only; 4xx and 5xx are never cached.
• ·model accepts a string or a non-empty array; an array drives the cost cascade.
• ·A cascade plus stream:true returns 400 fallback_with_stream_unsupported; a cascade plus idempotency-key returns 400 fallback_with_idempotency_unsupported.
• ·Webhook delivery is at-least-once; a sustained failure streak sets the endpoint to disabled_after_failures.

Related