Prompt caching

What it is and why it exists

Most multi-turn or RAG-style workloads resend the same opening bytes on every call. A long system prompt, a tool manifest, a few-shot block, a fixed document. You pay full input price for those bytes each time even though they never change. Prompt caching lets the provider keep an encrypted copy of that prefix and, on the next call that starts with the exact same bytes, charge you a fraction of the input rate to read it back instead of re-encoding it.

Ringside exposes this only on Anthropic-routed models, because the controls map directly onto Anthropic's cache_control breakpoint mechanism. The mental model is a marker you place on a message: "everything from the start of the request up to and including this message is a stable prefix, cache it." The first call that carries the marker writes the prefix to the cache (a one-time surcharge). Every later call within the TTL that begins with the identical prefix reads it back cheaply.

The two controls

You turn caching on one of two ways. Pick whichever fits how you build the request.

Explicit breakpoint: cache_control

Put a cache_control object on the message that ends your stable prefix. The shape is the Anthropic OpenAI-compat one: { "type": "ephemeral" }, optionally with a ttl of "5m" or "1h". Ringside validates this leniently. If type is anything other than "ephemeral", or ttl is anything other than "5m" or "1h", the hint is dropped silently rather than rejected, because a typo in a best-effort caching hint should not 400 an otherwise valid request.

There is also a top-level cache_control on the request body. If no message carries its own marker and the body has one, Ringside applies it to the last message in the array. That mirrors Anthropic's top-level cache_control on messages.create. A per-message marker always wins over the body-level one.

{
  "model": "fc:anthropic/claude-3-5-sonnet",
  "messages": [
    {
      "role": "system",
      "content": "<~4k tokens of stable instructions + schema + few-shot>",
      "cache_control": { "type": "ephemeral", "ttl": "1h" }
    },
    { "role": "user", "content": "First question" }
  ]
}

Automatic breakpoint: prompt_cache:"auto"

Set prompt_cache:"auto" on the request body and Ringside places a single breakpoint for you. It targets your last system message, so the cached prefix covers the tool manifest and the whole system block (more on ordering below). If there is no system message it falls back to the last message in the array. The default is "off".

Auto fires only when the value is "auto", the resolved model is Anthropic-backed and you have not already set a marker anywhere (neither a per-message cache_control nor a body-level one). If you set an explicit breakpoint, auto stays out of the way. So you can layer auto on a request and still override its placement by adding your own marker.

{
  "model": "fc:anthropic/claude-3-5-sonnet",
  "prompt_cache": "auto",
  "messages": [
    { "role": "system", "content": "<long stable system prompt>" },
    { "role": "user", "content": "Question one" }
  ]
}

Economics

Caching changes what each input token costs, not what it counts as. Three buckets, each priced as a multiple of the model's normal input rate.

The first call that carries a marker pays the 1.25x write surcharge on the cached prefix. That is the cost of populating the cache. Every subsequent call within the TTL that starts with the same bytes pays 0.10x on that prefix instead of 1.0x. Break-even comes fast. A prefix reused even two or three times within the window comes out well ahead of paying full price on every call.

The minimum cacheable prefix is about 4,000 tokens. A prefix shorter than that will not be cached even with a marker on it, so there is no point breaking a small system prompt at a cache point. The discount lands on the customer or developer wallet directly. Ringside meters the exact 3-bucket upstream cost, applies your markup on top and never pockets the cache delta. Read cost-control for how the markup and metering interact.

The usage fields that prove it

You do not have to trust that caching happened. The usage object on the chat completion carries two extra fields whenever the upstream reports a cache split.

cache_creation_input_tokens

Tokens written to the cache on this call. Non-zero on the first call that establishes a prefix (or any call that re-writes an expired one). This is the bucket billed at 1.25x.

cache_read_input_tokens

Tokens served from the cache on this call. Non-zero on later calls that share the prefix. This is the bucket billed at 0.10x, and it is mirrored into prompt_tokens_details.cached_tokens for OpenAI-SDK compatibility.

The pattern across a session is simple. The first call shows cache_creation_input_tokens set and cache_read_input_tokens at zero, then every following call within the TTL flips that, showing a populated cache_read_input_tokens and zero creation. Both fields are omitted entirely when neither is non-zero, so a plain uncached call carries the usual prompt_tokens / completion_tokens / total_tokens shape with nothing extra.

"usage": {
  "prompt_tokens": 4200,
  "completion_tokens": 120,
  "total_tokens": 4320,
  "cache_creation_input_tokens": 4096,
  "cache_read_input_tokens": 0,
  "prompt_tokens_details": { "cached_tokens": 0 }
}

Why ordering matters

A cached prefix is a byte-exact match. Anthropic renders a request in a fixed order: tools first, then the system block, then the messages. So a breakpoint placed on the system message captures the tool manifest and the system text in the cached prefix, which is exactly why auto mode targets the last system message rather than the last user message.

The practical rule. Keep everything stable at the front and let the volatile bytes trail after the breakpoint. Tools, system instructions, fixed few-shot examples, a pinned document. Those go before the marker. The per-call user question, freshly retrieved RAG chunks, a timestamp or a turn counter belong after it. If you splice a changing value into the prefix (a "current time" line in the system prompt, say) you break the byte match and turn every call into a fresh write at 1.25x instead of a read at 0.10x.

request render order  ->  [ tools ][ system ][ messages... ]
                                          ^
                          cache_control breakpoint here
                          (auto mode places it on the last system message)

cached prefix  = tools + system  (>= ~4,000 tokens to qualify)
volatile tail  = per-call user turn, RAG context, instruction nudges

Non-Anthropic upstreams

Both controls are no-ops on every non-Anthropic model, and that is by design so you can keep one request shape across providers. When you point a cache_control marker at, say, fc:openai/gpt-4o-mini, the OpenAI-compat adapter builds the upstream request by picking role and content (and tool fields) off each message and never copies the cache hint forward. The field is dropped before the request leaves Ringside. No 400, no warning, no behavior change.

Likewise prompt_cache:"auto" only fires its breakpoint when the resolved provider is Anthropic-backed, meaning either an fc:anthropic/* ref or a match:/slot:/dyn: ref that resolves to one. Against any other provider, auto resolves to no marker at all. That said, some providers do their own automatic context caching upstream (DeepSeek, for instance) and Ringside still meters those reads at the provider's published cache rate. You just do not control them with these two fields.

Worked example

A support assistant with a 4,200-token system prompt (policy text, a JSON output schema and three few-shot examples) answering a stream of short user questions. Mark the system message once with a 1-hour TTL. The first call writes the prefix; the rest read it.

1. 1
   First call: write the prefix

   Attribute it to a customer with the FC-Customer header. The response usage shows the write bucket populated and X-Ringside-Wallet-Deducted reflects the 1.25x surcharge on the cached portion.

   curl

   Copy

   curl https://api.fightclub.pro/v1/chat/completions \
     -H "Authorization: Bearer $FC_API_KEY" \
     -H "FC-Customer: cus_42" \
     -H "Content-Type: application/json" \
     -d '{
       "model": "fc:anthropic/claude-3-5-sonnet",
       "messages": [
         { "role": "system",
           "content": "<4,200 tokens of policy + schema + few-shot>",
           "cache_control": { "type": "ephemeral", "ttl": "1h" } },
         { "role": "user", "content": "What is my refund window?" }
       ]
     }'
   # usage.cache_creation_input_tokens > 0, cache_read_input_tokens == 0

2. 2
   Next calls: read the prefix

   Resend the identical system message (byte-for-byte) and change only the trailing user turn. Within the hour these read back at 0.10x.

   curl

   Copy

   curl https://api.fightclub.pro/v1/chat/completions \
     -H "Authorization: Bearer $FC_API_KEY" \
     -H "FC-Customer: cus_42" \
     -H "Content-Type: application/json" \
     -d '{
       "model": "fc:anthropic/claude-3-5-sonnet",
       "messages": [
         { "role": "system",
           "content": "<the SAME 4,200 tokens, unchanged>",
           "cache_control": { "type": "ephemeral", "ttl": "1h" } },
         { "role": "user", "content": "And for digital goods?" }
       ]
     }'
   # usage.cache_read_input_tokens > 0, cache_creation_input_tokens == 0

The same flow through the OpenAI SDK pointed at Ringside. The system prompt is the stable prefix; the cache fields come back on resp.usage.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.fightclub.pro/v1",
    api_key=os.environ["FC_API_KEY"],
)

SYSTEM = "<4,200 tokens of policy + schema + few-shot>"

def ask(question: str):
    resp = client.chat.completions.create(
        model="fc:anthropic/claude-3-5-sonnet",
        messages=[
            {
                "role": "system",
                "content": SYSTEM,
                "cache_control": {"type": "ephemeral", "ttl": "1h"},
            },
            {"role": "user", "content": question},
        ],
        extra_headers={"FC-Customer": "cus_42"},
    )
    u = resp.usage
    # these two fields are present only on Anthropic-routed cache hits
    print(getattr(u, "cache_creation_input_tokens", 0),
          getattr(u, "cache_read_input_tokens", 0))
    return resp.choices[0].message.content

ask("What is my refund window?")   # writes the prefix (1.25x once)
ask("And for digital goods?")      # reads the prefix (0.10x)

Failure modes and edge cases

Prefix under ~4,000 tokens

A marker on a prefix below the minimum does not cache. The call bills at full input price and usage carries no cache fields. Combine your stable bytes into one block above the threshold or do not bother marking it.

Drifting prefix bytes

Any change inside the cached span (a clock, a counter, reordered few-shot, whitespace edits) breaks the byte match. The next call writes a fresh prefix at 1.25x rather than reading at 0.10x. Keep volatile content strictly after the breakpoint.

TTL expiry

A read past the "5m" or "1h" window is a miss. The provider re-writes the prefix (you see cache_creation_input_tokens again) and the next in-window call reads it. Pick the TTL to match how tightly your calls cluster in time.

Malformed cache_control

A type other than "ephemeral", or a ttl other than "5m"/"1h", is dropped silently, not rejected. The request still runs, just without caching. There is no cache_control-specific error code.

Wrong provider

Pointing the controls at a non-Anthropic model is silently a no-op. If you expected a cache and usage shows none, confirm the resolved model is Anthropic-backed via the X-Ringside-Model-Resolved response header.

Streaming without include_usage

A streamed response omits the usage object unless you pass stream_options: { include_usage: true }. Without it you cannot read the cache split, even though caching still happened.

Defaults and limits at a glance