Going to production

What this page is for

A Ringside integration that works in a Postman tab and one that works under sustained production load are different things. The wire calls are identical. What changes is everything around them: where your secret lives, how the browser talks to the API, what happens when a provider 503s mid-stream, whether a network retry creates a duplicate charge and whether you can find the one failing request out of a million in your logs.

This checklist is ordered roughly by blast radius. The first few items (key handling, client tokens) are the ones that turn into a security incident if you get them wrong. The middle (budgets, rate limits, error handling, idempotency) is what keeps spend and reliability sane. The last few (regions, observability, load testing) are what let you operate the thing once it is live. Each item links to the deeper page where the mechanics live.

The checklist

1. 01Keys are server-side only and rotated. The ko_<64 hex> developer key lives in your backend env, never in client code, never in a repo. You can rotate by minting a new key in the dashboard and revoking the old one.
2. 02Browser traffic uses client tokens, origin-locked and short. If a frontend calls Ringside directly, it sends Authorization: Client <jwt>, never your ko_ key. Mint each token server-side with a tight ttl_seconds, an origin_allowlist and (where you can resolve a stable client IP) bind_ip.
3. 03Budgets and wallets are set per customer. Every customer that should have a spend ceiling carries a monthly_budget_usd, a prepaid wallet or both. Your own developer wallet has a balance and a wallet.low webhook wired so it never silently empties.
4. 04Rate limits are chosen, not defaulted. Pick a per-customer rpm_limit / tpm_limit and a per-client-token rpm_limit that match real usage, so one tenant or one leaked token cannot exhaust the shared ceiling.
5. 05The webhook receiver verifies signatures and is idempotent. It recomputes the X-FC-Signature HMAC over the raw body, constant-time compares, rejects stale timestamps and de-dupes on the event id so a retried delivery is processed once.
6. 06Error handling branches on code with backoff. Your client switches on the stable error.code, never the human message, then honours Retry-After on a 429 with capped exponential backoff and jitter.
7. 07Writes carry an idempotency-key. Any non-idempotent create (a customer, an assistant, a run, a one-shot chat you must not double-bill) sends the lowercase idempotency-key header so a network retry replays instead of re-executing.
8. 08Region is pinned if residency matters. If you have an EU or US data-residency obligation, your fc: model refs carry the matching @eu / @us (or Bedrock / Vertex) suffix and you have confirmed the resolved model honoured it.
9. 09Observability is wired. You capture X-Request-Id on every response (success and failure), log the error.code on failures and have at least the request.failed_after_retry, wallet.low and customer.budget_exceeded webhooks landing somewhere a human will see them.
10. 10You load tested against the real ceilings. You have driven expected peak concurrency at staging and watched X-RateLimit-Remaining and wallet draw-down, so launch day is not the first time you see a 429.

Keys: server-side only, rotated

A developer key is literally ko_ followed by 64 hex characters, sent as Authorization: Bearer ko_.... It is SHA-256 hashed at rest, so the dashboard can show you the prefix but never the full key again after creation. Treat it like a database password. It belongs in a secret manager or your platform env, injected at runtime, nowhere a git grep or a browser devtools Network tab can reach it.

• ·One key per environment. Staging and production hold different keys so revoking one does not take down the other.
• ·Rotate by overlap, not by gap: create the new key, deploy it, confirm traffic flows on it, then revoke the old one in the dashboard at /app/api-keys.
• ·Scope matters for one thing in particular: minting client tokens needs a key holding the api:client_tokens scope. A key without it gets 403 insufficient_scope when it calls the mint endpoint.
• ·A leaked key is a revoke-and-rotate, not a password reset. There is no per-request signing you can fix instead.

Client tokens: origin-locked and short

When a frontend talks to Ringside directly, mint a client token server-side and hand only that to the browser. The browser then authenticates with the Client scheme, not Bearer: Authorization: Client <jwt>. That scheme is precisely what marks the request as a scoped client token rather than a developer key. The Authorization parser accepts only Bearer or Client ; anything else is 401 with the message that the header must start with one of those two.

Mint at POST /v1/customers/{id}/client_tokens (the {id} accepts ext:<external_id> too). Spend lands on the customer in the path. Tune four knobs so a leaked token is close to worthless:

Minting is rate-limited to 100/min per customer, so do not mint a fresh token on every keystroke. Mint once per session, cache it client-side until expires_at and re-mint on expiry. On logout or a suspected leak, revoke: DELETE .../client_tokens/{jti} for one (the jti is prefixed ctok_) or POST .../client_tokens/revoke_all to kill every active token for the customer.

# backend: mint a 10-minute, origin-locked, rate-limited token
curl https://api.fightclub.pro/v1/customers/cus_42/client_tokens \
  -H "Authorization: Bearer $FC_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "ttl_seconds": 600,
    "scope": ["chat"],
    "rpm_limit": 20,
    "origin_allowlist": ["https://app.example.com"]
  }'
# => { "token":"...", "jti":"ctok_...", "expires_at":"...", "customer_id":"cus_42" }

# browser: note the "Client" scheme, and the Origin must match the allowlist
curl https://api.fightclub.pro/v1/chat/completions \
  -H "Authorization: Client <token>" -H "Origin: https://app.example.com" \
  -d '{"model":"fc:openai/gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'

Budgets and wallets per customer

Customers are first-class (cus_). They are how you stop one tenant running up an unbounded bill and how you attribute spend. Attribute a metered call with the FC-Customer header (a cus_ id, ext:<external_id> or a plain external id string). Two independent guards exist and they return different status codes, so handle both.

monthly_budget_usd

A monthly spend ceiling. When attributed spend would exceed it, the call returns 402 customer_budget_exceeded. Your developer wallet still has money; this is a per-customer cap, not an account-level one.

Prepaid wallet

A balance you top up via POST /v1/customers/{id}/wallet/topup and adjust via .../wallet/adjust. An empty customer wallet returns 402 customer_wallet_empty. Use this for prepaid / pay-as-you-go tenants.

Developer wallet

Your own balance. Every unattributed call (or attributed call without a customer wallet) draws from it. Empty returns 402 wallet_empty. Read X-Ringside-Wallet-Balance and X-Ringside-Wallet-Deducted on each response to track it.

Set FC-Customer-Strict: true on calls where you do not want an unknown id to lazily create a new customer. Without it, a typo in FC-Customer quietly mints a fresh customer rather than erroring. For resale, set default_billable_markup_pct or default_billable_amount_usd on the customer (they are mutually exclusive; setting both is 400 billable_defaults_mutex) and read your margin at GET /v1/customers/{id}/margin.

Rate limits: choose them

Ringside enforces rate limits at four layers and the lowest applicable ceiling wins: edge per-IP, per-client-token, per-developer, then per-customer. You set the middle two. Leaving everything at defaults means one noisy tenant or one leaked token can soak up the shared budget and 429 everyone else.

On every response, read X-RateLimit-Remaining to see how much window is left. On a 429, the response carries Retry-After as an integer number of seconds. There is no X-RateLimit-Limit and no X-RateLimit-Reset header, so do not write code that depends on them. A customer-level cap hit returns 429 customer_rate_limit_exceeded; the per-IP / per-token / per-dev ceilings return 429 rate_limited. Branch on the code, honour Retry-After.

Error handling: branch on code, back off

Every error has the same body shape. type, code and message are always present; param appears on field-validation errors and retry_after_seconds appears on rate-limit errors. The message is for humans and can change. Always switch on the stable code. The request id is the X-Request-Id response header, never a body field.

{
  "error": {
    "type": "rate_limit_error",
    "code": "customer_rate_limit_exceeded",
    "message": "Customer rpm cap reached.",
    "retry_after_seconds": 12
  }
}

The nine type values group the codes: authentication_error, invalid_request_error, permission_error, rate_limit_error, conflict_error, not_found_error, wallet_error, internal_error and api_error. Build your retry policy off the status family plus the code, not off the prose.

Retry the transient classes only: 429 and 503 upstream_unavailable. Use capped exponential backoff with jitter, and when Retry-After (or retry_after_seconds) is present, treat it as a floor on your wait. A model cascade (passing model as an array) handles provider flakiness for you: Ringside walks the chain on 5xx and surfaces the winner via X-Ringside-Model-Used. Note the cascade cannot combine with stream:true or an idempotency-key.

Idempotency on writes

Networks retry. A POST that times out on your side may well have succeeded on ours, and a naive retry creates a second customer, a second run or a second charge. The fix is one header, exactly idempotency-key (lowercase), carrying a unique value per logical operation. Replay it inside the 24 h window and Ringside returns the original result instead of re-executing. Scope is (developer, customer, key), and it works the same on every endpoint that mutates.

KEY=$(uuidgen)
# first send creates the run; a timed-out retry with the same key replays it
curl https://api.fightclub.pro/v1/threads/thread_abc/runs \
  -H "Authorization: Bearer $FC_API_KEY" \
  -H "idempotency-key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{"assistant_id":"asst_xyz"}'

Pin the region if residency matters

If you have an EU or US data-residency obligation, the model ref carries it. The @<region> suffix attaches to fc: refs only and routes the call to a provider deployment in that region. An unknown or unsupported suffix is a 400. Confirm the routing took effect by reading X-Ringside-Model-Resolved on the response.

# EU-pinned OpenAI
-d '{"model":"fc:openai/gpt-4o-mini@eu", ... }'
# EU-pinned Claude via Bedrock
-d '{"model":"fc:anthropic/claude-sonnet-4@eu-bedrock", ... }'

Observability

You cannot operate what you cannot see. The one identifier worth logging on every single call, success or failure, is the X-Request-Id response header (request ids are prefixed req_). Quote it in support tickets and it points us straight at the call. Log the error.code alongside it on failures so you can aggregate by failure class rather than by free-text message.

• ·Capture X-Request-Id on every response and attach it to your own log line for the call.
• ·On failures, log error.type and error.code; alert on spikes in any single code.
• ·Track wallet draw-down from X-Ringside-Wallet-Balance / X-Ringside-Wallet-Deducted so a slow leak shows up before 402 wallet_empty does.
• ·On a model cascade, log X-Ringside-Models-Tried and X-Ringside-Fallback-Triggered so you can see how often your cheap-first model is failing over.
• ·Subscribe an ops webhook to request.failed_after_retry, wallet.low, customer.budget_exceeded and moderation.flagged at minimum.

Verify every webhook delivery

Your webhook receiver is an inbound API that anyone on the internet can POST to. Treat it as hostile until verified. Each genuine delivery carries X-FC-Signature: t=<unix>,v1=<hex>, where v1 is the HMAC-SHA256 of the raw request body keyed by the endpoint secret (the whsec_ value returned once on create and on rotate_secret). Verify the raw body before you parse it, compare constant-time and reject deliveries whose t= timestamp is too old to block replays.

import hashlib, hmac, time

def verify_and_handle(raw_body: bytes, sig_header: str, secret: str, seen: set):
    parts = dict(p.split("=", 1) for p in sig_header.split(","))
    ts = int(parts["t"])
    if abs(time.time() - ts) > 300:          # reject stale (>5 min)
        raise ValueError("stale timestamp")

    signed = f"{ts}.".encode() + raw_body
    expected = hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
    # during a rotation grace the header can carry two v1= digests; accept either
    candidates = [v for k, v in (p.split("=", 1) for p in sig_header.split(",")) if k == "v1"]
    if not any(hmac.compare_digest(expected, c) for c in candidates):
        raise ValueError("bad signature")

    event = parse_json(raw_body)
    if event["id"] in seen:                  # idempotent: a retry lands once
        return
    seen.add(event["id"])
    process(event)

Make the handler idempotent on the event id, because deliveries retry with backoff and you will receive the same event more than once. A sustained failure streak flips the endpoint to disabled_after_failures and you have to re-enable it (PATCH status=active), so return a 2xx fast and do the slow work off the request path. The POST /v1/webhooks/{id}/test route and the GET .../deliveries log let you exercise this end to end before you depend on it.

Load test against the real ceilings

Launch day should not be the first time your integration meets a 429 or watches a wallet drain. Drive expected peak concurrency at a staging customer with a small budget and a known wallet balance, then watch the headers. Confirm that your backoff actually fires on a 429 and recovers, that X-Ringside-Wallet-Deducted matches what you expect to be billing and that a customer.budget_exceeded at the cap surfaces the 402 cleanly rather than as an unhandled exception.

• ·Run at and slightly above your chosen per-customer rpm_limit to confirm the 429 path and Retry-After handling work.
• ·Drain a test customer wallet to zero to confirm your 402 customer_wallet_empty handling, then top it back up.
• ·Fire a burst of identical writes with the same idempotency-key to confirm exactly one effect.
• ·If you use region pinning, assert X-Ringside-Model-Resolved under load, not just in a single happy-path call.

One last pass before you flip the switch

Run the ten-item checklist once more against the real production config, not the staging copy. The two that bite hardest are the secret in the wrong place and the unverified webhook, because both look fine until someone exploits them. Everything else is recoverable in minutes; those two are incidents.