Observability

Why this matters

You are running someone else's inference through your account. When a customer reports a slow response, a wrong answer or a surprise bill, you need to know what actually happened on that one call. Ringside answers that at four levels. The response itself carries a request id and a set of metering headers. The request log keeps a per-call row you can pull back by id. The usage and margin endpoints aggregate those rows into cost and revenue rollups. And webhooks push the events you cannot afford to poll for.

Wire all four. The response headers give you live correlation, the log gives you forensic depth, the rollups give you trends and the webhooks give you alerts. Skip any one of them and you end up reconstructing state you could have captured for free.

one call
   |
   |--> response headers   X-Request-Id, X-Ringside-Model-Used, X-Ringside-Wallet-Deducted ...
   |--> request log row    GET /v1/requests/{id}   (full request + response preview, tokens, cost, status)
   |--> usage rollup       GET /v1/usage?group_by=...   (cost + tokens by day/model/customer/tag/property)
   |--> margin rollup      GET /v1/customers/{id}/margin   (billable - cost)
   '--> webhook event      request.failed_after_retry, wallet.low, customer.budget_exceeded, batch.* ...

Capture the request id on every call

The correlation id is the X-Request-Id response header. It is never a body field. Log it on success and on failure, store it next to your own trace id and put it in any support ticket. With it you can pull the exact log row back later, and you can quote it to us if you need to.

curl -sS -D - -o /dev/null https://api.fightclub.pro/v1/chat/completions \
  -H "Authorization: Bearer ko_<key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"fc:openai/gpt-4o-mini","messages":[{"role":"user","content":"ping"}]}' \
  | grep -i '^x-request-id:'

# x-request-id: req_9f3c1a77b2e40af1

The request log

Every call writes one row to the request log. Pull a single row by id with GET /v1/requests/{id}, or page the list with GET /v1/requests. The single-record fetch returns full request and response previews; the list view truncates both previews to 200 characters to keep the payload small. Cross-tenant access returns 404, not 403, so ids stay non-enumerable.

In the dashboard the same data is at ringside.fightclub.pro/app/logs, with a per-request detail view at /app/logs/{request_id}. Paste a req_ id from a customer report straight into that path.

Fields on a log row

Filters on the list

GET /v1/requests accepts customer_id, conversation_id, session_id, endpoint, status, model (matches either requested or resolved), tag, property_key + property_value (both required together) and a from / to date window. Paginate with limit (default 20, max 100) and after (pass the previous next_cursor). The list returns { "data": [...], "has_more": boolean, "next_cursor": string | null }. There is no object: "list" field.

curl -sS "https://api.fightclub.pro/v1/requests?customer_id=cus_42&session_id=checkout-7f2&status=error&from=2026-06-23T00:00:00Z&limit=50" \
  -H "Authorization: Bearer ko_<key>"

# {
#   "data": [ { "id": "req_9f3c1a77b2e40af1", "status": "error",
#               "error_type": "wallet_error", "model_resolved": "gpt-4o-mini",
#               "duration_ms": 812, "retry_count": 2, ... } ],
#   "has_more": false,
#   "next_cursor": null
# }

Metering headers

On a chat completion the response carries the billing and routing outcome inline, so you can record it without a second lookup. Two headers report money. X-Ringside-Wallet-Balance is the wallet balance after the call and X-Ringside-Wallet-Deducted is the exact charge, both to 7 decimal places. When the call is attributed to a customer wallet, these report that wallet; otherwise they report your developer wallet.

There is no X-RateLimit-Limit and no X-RateLimit-Reset. On a 429, read Retry-After (integer seconds) instead. Rate limiting has four layers and the lowest wins: edge per-IP, per-client-token, per-developer and per-customer. A 429 from any layer looks the same on the wire.

Usage and margin rollups

For trends, not single calls, aggregate the log. GET /v1/usage rolls up cost and tokens over a from / to window (default last 30 days). Set group_by to day, model, customer, tag or property. With group_by=property you must also pass property_key. Add customer_id to scope to one customer, or tag to filter to a single tag before aggregating.

The response carries a total block (cost, input and output tokens, cached and reasoning tokens, request count, message count, conversation count) plus a buckets array, each bucket a { key, llm_cost_usd, input_tokens, output_tokens, request_count }. An invalid group_by returns 400 invalid_group_by; a missing property_key returns 400 missing_property_key; an unparseable date returns 400 invalid_date with the offending param.

curl -sS "https://api.fightclub.pro/v1/usage?group_by=model&from=2026-06-01T00:00:00Z" \
  -H "Authorization: Bearer ko_<key>"

# {
#   "total": { "llm_cost_usd": 412.88, "input_tokens": 19284113,
#              "output_tokens": 3120554, "request_count": 88241, ... },
#   "buckets": [
#     { "key": "gpt-4o-mini", "llm_cost_usd": 201.40, "request_count": 61002, ... },
#     { "key": "claude-sonnet-4", "llm_cost_usd": 187.10, "request_count": 24109, ... }
#   ],
#   "from": "2026-06-01T00:00:00.000Z", "to": "...", "group_by": "model"
# }

Margin is revenue minus cost, and it lives on the customer. GET /v1/customers/{id}/margin returns cost_charged_usd, billable_amount_usd, margin_usd and margin_pct (null when revenue is zero, to avoid a noisy 0/0) per bucket and in total. group_by is day, customer or model. Only status=ok calls contribute; blocked and errored calls carry neither revenue nor cost. An unknown customer id returns 404, same as everywhere else.

curl -sS "https://api.fightclub.pro/v1/customers/cus_42/margin?group_by=day&from=2026-06-01T00:00:00Z" \
  -H "Authorization: Bearer ko_<key>"

# {
#   "customer_id": "cus_42",
#   "total": { "cost_charged_usd": 18.40, "billable_amount_usd": 31.00,
#              "margin_usd": 12.60, "margin_pct": 40.64, "request_count": 4120 },
#   "buckets": [ { "key": "2026-06-23", "margin_usd": 1.92, "margin_pct": 41.2, ... } ],
#   "group_by": "day"
# }

Operational webhooks

Some conditions you cannot poll for fast enough. Register an HTTPS endpoint with POST /v1/webhooks and subscribe to the operational slice of the taxonomy. The events below are the ones an on-call rotation actually wants to wake up for.

wallet.low / wallet.empty

Your developer wallet is running down or out. Top up before calls start failing with 402 wallet_empty.

customer.budget_exceeded

An attributed customer crossed its monthly_budget_usd; further calls for that customer return 402 customer_budget_exceeded.

customer.rate_limit_exceeded

A customer hit its rpm_limit / tpm_limit. Pairs with the 429s you will see on that customer's traffic.

request.failed_after_retry

A call exhausted its retry budget and gave up. This is your upstream-degradation signal.

batch.created / batch.completed / batch.failed / batch.cancelled

Async batch lifecycle, so you act on completion instead of polling GET /v1/batches/{id} on a timer.

moderation.flagged

Inline moderation flagged content on a chat call that used the moderation param.

Delivery carries X-FC-Signature: t=<unix>,v1=<hex>, where v1 is the HMAC-SHA256 of the raw request body keyed by the endpoint secret (whsec_, returned once on create and on rotate_secret). Verify constant-time, reject stale timestamps and, during a secret rotation, accept either of the two v1= digests in the header. The same evt_ id rides every retry and every parallel fan-out, so dedupe on it. A sustained delivery-failure streak auto-disables the endpoint (status: disabled_after_failures); watch for that, because a dead endpoint means missed alerts.

A logging checklist

1. 01Record X-Request-Id on every response, success and failure, next to your own trace id.
2. 02On error, log error.type and error.code from the body, and branch on code, never message.
3. 03Record X-Ringside-Model-Used and X-Ringside-Fallback-Triggered so a silent fallback to a pricier model is visible.
4. 04Treat the per-call X-Ringside-Wallet-* headers as live signal, but reconcile spend against cost_charged_usd on the log row or usage rollup, which are written transactionally.
5. 05Attach FC-Session-Id, FC-Tag and FC-Property-* at call time so usage rolls up by the dimensions you care about; you cannot add them after the fact.
6. 06Subscribe to wallet.low, customer.budget_exceeded and request.failed_after_retry, verify the signature constant-time and alert on disabled_after_failures.
7. 07Run a daily GET /v1/usage?group_by=model and per-customer margin pull to catch cost drift before the invoice does.