Rate limiting and quotas

Why four layers exist

Ringside sits between your application and 19 upstream providers, and it bills against a wallet on every call. Rate limiting has to protect the shared edge from a single noisy source IP, your developer account from runaway spend and each of your customers from the others all at once. A single per-account limit cannot express that. So enforcement is split into four independent layers, evaluated in order, and the first layer that says no is the one that returns the 429.

The layers are not additive and they do not negotiate. Each holds its own counter. When you mint a client token with rpm_limit: 600 for a customer whose rpm_limit is 120, the token is clamped to 120 at mint time, and even then the per-customer layer can still reject the request if other tokens for the same customer have already filled the window. Think of it as the minimum of every limit that applies to a given request.

The four enforcement layers

RPM versus TPM

The per-customer layer is the one you configure directly, and it has two dimensions. RPM caps requests per minute. TPM caps tokens per minute. They are set as rpm_limit and tpm_limit on the customer record, and either can be left null to disable that dimension. A value that is null or less than or equal to zero is treated as off.

rpm_limit

Requests per minute for this customer, enforced as an in-memory sliding window of request timestamps over the trailing 60 seconds. A request is counted the moment it passes the check, so the check is a commit, not a preview.

tpm_limit

Tokens per minute for this customer, enforced as a rolling 60-second SUM of input_tokens + output_tokens over completed (status ok) usage events, compared against that running total plus the estimated input and output tokens of the incoming request.

The two windows differ in a way that matters when you reason about retries. RPM is a true sliding window held in process: each accepted request appends a timestamp, and the window is the last 60 seconds at the instant of the check. TPM is computed from the usage ledger, so it only sees tokens from requests that already finished and recorded status = 'ok'. An in-flight burst of large completions can therefore pass the TPM check on the way in and only push you over once they settle.

How TPM estimates the incoming request

Before dispatch, Ringside adds the estimated input and output tokens of the new request to the rolling 60-second sum. If used + estimated > tpm_limit, the request is rejected with limit_type tpm. Because the estimate is added pre-flight, a single request whose estimated tokens alone exceed the remaining budget will be blocked outright rather than partially admitted.

Response headers

Ringside advertises remaining request budget on the response, but it is deliberately minimal. There is exactly one steady-state header and one extra header on a 429. Casing is exact and must be matched case-insensitively by your client (HTTP header names are case-insensitive, but the wire casing below is what Ringside emits).

One subtlety: X-RateLimit-Remaining is omitted entirely when the customer has no rpm_limit set, rather than sent as some sentinel like -1. Code that reads the header must treat absence as unlimited, not as zero.

Client-token RPM defaults and the ceiling

Client tokens carry their own RPM, set at mint time via POST /v1/customers/{id}/client_tokens. If you omit rpm_limit, the token gets 60 requests per minute. The hard ceiling is 600, and on top of that the value is clamped to the customer's rpm_limit if one is set.

At request time the per-token limiter runs its own sliding 60s window keyed by the token jti. If the registry row somehow has no usable limit, the limiter falls back to 60 rather than letting traffic through unbounded. So a missing limit never disables the layer.

What a 429 looks like

Every rate-limit rejection uses the standard Ringside error envelope. type, code and message are always present, and rate-limit errors add retry_after_seconds. Always branch on code, never on the human-readable message. The request id is in the X-Request-Id response header, never in the body.

{
  "error": {
    "type": "rate_limit_error",
    "code": "customer_rate_limit_exceeded",
    "message": "Customer rpm limit exceeded",
    "retry_after_seconds": 17
  }
}

Note that retry_after_seconds in the body and the Retry-After response header carry the same integer. The body field is convenient when you have already deserialized the error; the header is convenient in middleware that never reads the body. Either is authoritative.

The three rate-limit codes

Handling a 429 correctly

The rule is short. On a 429, sleep for the seconds Ringside gives you, then retry. Do not invent your own constant backoff, and do not retry immediately. The server computes retry_after_seconds as the time until the oldest counted request leaves the window (for RPM) or a safe 60 seconds (for the TPM rolling sum), so honouring it lands you exactly at the moment a slot frees up.

1. 01Read the status. A 429 means rate-limited; a 402 (customer_budget_exceeded, customer_wallet_empty, wallet_empty) means out of money, which is not retryable without topping up.
2. 02Read Retry-After from the response header (integer seconds), or retry_after_seconds from the error body. They match.
3. 03Sleep for that many seconds, with a small jitter if you have many workers, to avoid a thundering herd at the same instant.
4. 04Retry the same request. If you are using an idempotency-key, the retry is safe and will not double-charge within the 24h replay window.
5. 05Cap retries. After a few attempts, surface the error to the caller rather than looping forever.

curl -i https://api.fightclub.pro/v1/chat/completions \
  -H "Authorization: Bearer ko_<key>" \
  -H "FC-Customer: cus_42" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "fc:openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "ping"}]
  }'

# HTTP/1.1 429 Too Many Requests
# X-Request-Id: req_8f31c0a2b9e44d1a
# X-RateLimit-Remaining: 0
# Retry-After: 17

Edge cases worth knowing

• ·A client token minted with rpm_limit above the customer cap is rejected at mint time with 400 rpm_exceeds_customer_cap, not silently clamped. Clamping to the customer rpm is something you should do in your mint call, not rely on the server to swallow.
• ·TPM can reject a request that RPM allowed, and vice versa. They are independent dimensions of the same per-customer layer; either one tripping returns customer_rate_limit_exceeded.
• ·Because TPM reads the usage ledger, a burst of large concurrent completions can each pass the inbound check and only collectively push the next request over. Size your tpm_limit with headroom for in-flight work.
• ·X-RateLimit-Remaining reflects only the per-customer RPM window. If a different layer (edge, developer or client-token) is what rejects you, the remaining header you saw on the prior success does not predict that.
• ·A 402 is not a rate limit. customer_budget_exceeded, customer_wallet_empty and wallet_empty mean spend, not throughput, and retrying without topping up will fail again.

Defaults and limits at a glance

Related