Enforce structured output

Why this exists

Structured output is the boring-but-load-bearing part of putting an LLM in a pipeline. Your downstream code expects an object with known fields. If the model wraps its answer in prose, fences it in a Markdown block or drops a required field, your JSON.parse throws and the whole job fails. OpenAI gpt-4o and Google Gemini can enforce a JSON schema at the provider level. Anthropic Claude, most open-weight models and anything self-hosted cannot.

Ringside flattens that difference. You always send the same response_format. When the resolved model supports json_schema natively, Ringside passes the schema straight through to the provider and lets it enforce. When the model does not, Ringside enforces the schema itself at runtime: it compiles your schema with ajv, instructs the model to return only conforming JSON, then parses and validates the reply. A non-conforming reply is re-prompted with the exact validation errors and retried. Two parameters, one contract. Your client code does not branch on provider.

The mental model

One request flows down one of two paths, chosen by the resolved model's capability. You do not choose the path. Ringside looks up the model in its capability matrix and decides.

POST /v1/chat/completions
  response_format: {type: json_schema, json_schema: {...}}
        |
        v
  resolve model ref  ->  capability lookup
        |
   supports_json_schema ?
        |                         |
       yes                        no
        |                         |
  pass schema to            runtime enforcement (ajv)
  provider, it              compile -> prompt -> generate
  enforces natively         -> strip fences -> parse -> validate
        |                         |
        |                  valid? --yes--> return JSON
        |                         |
        |                        no, re-prompt with ajv errors
        |                         | (up to 2 retries, 3 attempts total)
        |                         |
        |                  still invalid --> 422
        v                         v
     200 OK                response_schema_validation_failed

The split matters for two reasons. The first is latency. The runtime loop can call the upstream up to three times for one logical request, so a non-native model under a strict schema is slower in the worst case than a model that enforces the schema in a single provider call. The second is streaming. The runtime loop has to see the whole reply before it can validate, so streaming with a schema on a non-native model is rejected up front. Both are covered below.

Native support, by provider

Whether a model is native is a property of the underlying model, not the region or cloud, so a @eu or @us-bedrock residency suffix does not change it. Models not in the curated matrix default to non-native (safe-by-default: they run the ajv loop). The current curation:

The runtime loop in detail

When the resolved model is non-native, Ringside extracts the inner json_schema.schema and runs this sequence. None of it is visible on the wire on success: you get a normal 200 with a chatcmpl- completion whose content is the validated JSON string.

1. 01Compile your schema with ajv (non-strict mode, with ajv-formats so format keywords like email and date-time work). Compiled validators are cached by a SHA-256 hash of the schema JSON, so repeated calls with the same schema skip the recompile.
2. 02Prepend a system message instructing the model to return a single JSON object matching the schema EXACTLY, with no prose and no Markdown fences. The stringified schema is inlined into that same message.
3. 03Call the adapter.
4. 04Strip Markdown code fences from the reply (json ... or plain ... ), because models add them even when told not to.
5. 05Parse the stripped text as JSON, then run the compiled ajv validator.
6. 06If it parses and validates, return it. The returned content is the re-serialized parsed object, so what your client receives is canonical JSON.
7. 07If parsing fails or ajv returns false, prepend a reinforcement system message (carrying the exact ajv error objects on a validation miss, or a plainer "that was not valid JSON" note on a parse miss) and retry.

The retry budget is two re-prompts, so three upstream calls at most for one request. That default is fixed for the chat endpoint. A model that parses-but-fails-validation on attempt one gets the ajv errors verbatim on attempt two, which is usually enough for a capable model to correct a missing field or a wrong enum value.

Compile cache

Up to 128 compiled validators, keyed by schema hash, evicted LRU. Re-using one schema across many requests is cheap.

Fence stripping

Applied to every reply before parsing. A model that answers with a fenced block still validates.

Re-serialization

On success Ringside returns JSON.stringify(parsed), not the raw model text, so trailing whitespace or fences never reach you.

Retry count

Number of re-prompts that were needed (0, 1 or 2). 0 means it conformed first try.

The wire shape

Send response_format in the chat body exactly as OpenAI defines it. The inner json_schema object carries name, your schema and an optional strict. Ringside reads json_schema.schema for the runtime loop; native providers receive the whole response_format object and apply their own rules.

{
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "support_ticket",
      "schema": {
        "type": "object",
        "additionalProperties": false,
        "properties": {
          "category": { "type": "string", "enum": ["billing", "bug", "feature_request", "other"] },
          "severity": { "type": "integer", "minimum": 1, "maximum": 5 },
          "summary":  { "type": "string", "maxLength": 280 },
          "needs_human": { "type": "boolean" }
        },
        "required": ["category", "severity", "summary", "needs_human"]
      }
    }
  }
}

Worked example

Classify an inbound support message into the schema above, attributed to customer cus_42. The model here is fc:anthropic/claude-haiku-4-5, which is non-native, so this exercises the runtime loop. The same body against fc:openai/gpt-4o-mini would take the native path with no code change.

curl https://api.fightclub.pro/v1/chat/completions \
  -H "Authorization: Bearer ko_<your-64-hex-key>" \
  -H "Content-Type: application/json" \
  -H "FC-Customer: cus_42" \
  -d '{
    "model": "fc:anthropic/claude-haiku-4-5",
    "messages": [
      {"role": "system", "content": "Classify the support message."},
      {"role": "user", "content": "I was charged twice this month and the refund button 500s."}
    ],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "support_ticket",
        "schema": {
          "type": "object",
          "additionalProperties": false,
          "properties": {
            "category": {"type": "string", "enum": ["billing", "bug", "feature_request", "other"]},
            "severity": {"type": "integer", "minimum": 1, "maximum": 5},
            "summary": {"type": "string", "maxLength": 280},
            "needs_human": {"type": "boolean"}
          },
          "required": ["category", "severity", "summary", "needs_human"]
        }
      }
    }
  }'

On a 200 the content is canonical JSON Ringside re-serialized after validating, so json.loads (or JSON.parse) is safe. You do not need a tolerant parser or a fence stripper of your own. The one case where the parse would be on untrusted text is if you ignore the status code and parse a 422 body as if it were a completion, so branch on status first.

When the model never conforms: 422

If the model still fails after the two re-prompts, the call does not silently return garbage. Ringside returns HTTP 422 with code response_schema_validation_failed. The wallet is not charged for the failed completion (the usage event is recorded with zero cost and status: error), and the response carries everything you need to debug.

{
  "error": {
    "type": "invalid_request_error",
    "code": "response_schema_validation_failed",
    "message": "Model response did not validate against provided json_schema after 2 retries.",
    "retry_count": 2,
    "reason": "schema_validation_failed",
    "validation_errors": [
      {
        "instancePath": "/severity",
        "schemaPath": "#/properties/severity/maximum",
        "keyword": "maximum",
        "params": { "limit": 5 },
        "message": "must be <= 5"
      }
    ],
    "last_content": "{\"category\":\"billing\",\"severity\":9,\"summary\":\"double charge\",\"needs_human\":true}"
  }
}

Handling the 422 in code

import requests

r = requests.post(
    "https://api.fightclub.pro/v1/chat/completions",
    headers={
        "Authorization": "Bearer ko_<your-64-hex-key>",
        "FC-Customer": "cus_42",
        "Content-Type": "application/json",
    },
    json=body,  # the body from the worked example
    timeout=60,
)

request_id = r.headers.get("X-Request-Id")  # never in the body; always the header

if r.status_code == 200:
    ticket = r.json()["choices"][0]["message"]["content"]
elif r.status_code == 422 and r.json()["error"]["code"] == "response_schema_validation_failed":
    err = r.json()["error"]
    # err["reason"] tells you parse-vs-validate; err["validation_errors"] is the ajv detail.
    raise RuntimeError(f"schema enforcement failed ({err['reason']}) req={request_id}")
else:
    r.raise_for_status()

Read the request id from the X-Request-Id response header, never from the body. Log it alongside the failure: it is the id you quote when you open a support ticket or look the call up at /app/logs.

The cascade: let a stronger model rescue the schema

A response_schema_validation_failed is one of the escalation triggers in the model cascade. Pass model as an array and Ringside walks it in order: if the first model exhausts its re-prompts and would 422, the cascade tries the next model in the chain instead of returning the error. The 422 only surfaces if every model in the chain fails.

{
  "model": ["fc:anthropic/claude-haiku-4-5", "fc:openai/gpt-4o-mini"],
  "messages": [ ... ],
  "response_format": { "type": "json_schema", "json_schema": { "name": "support_ticket", "schema": { ... } } }
}

On success the response reports the chain via X-Ringside-Model-Used, X-Ringside-Models-Tried and X-Ringside-Fallback-Triggered. There is one hard restriction: a cascade plus an idempotency-key is rejected 400 fallback_with_idempotency_unsupported, because the cached entry would be the first attempt's failure and would mask the eventual success on replay.

Constraints and limits

No streaming with a schema on a non-native model

The runtime loop must see the entire reply before it can strip fences, parse and validate, so it cannot stream. If you set stream: true and response_format.type === "json_schema" on a model that does not support json_schema natively, the request is rejected before dispatch with 400 streaming_with_schema_fallback_unsupported. Set stream: false, or pick a model that enforces natively (gpt-4o family, Gemini), in which case streaming a schema is allowed because the provider does the enforcing.

Practical guidance

• ·Prefer a native model (gpt-4o-mini, Gemini Flash) for hot structured-output paths. The schema is enforced by the provider in one call, no re-prompt latency.
• ·Use a cascade (cheap-non-native first, native second) when cost matters more than worst-case latency: the cheap model handles the easy cases, the native model rescues the hard ones.
• ·Always branch on the HTTP status before parsing the body. A 200 body is canonical JSON; a 422 body is an error envelope.
• ·Branch on error.code, not error.message. The message wording can change; the code is stable.
• ·Set additionalProperties: false and a full required list. Open schemas let weak models add stray keys that still validate but pollute your data.
• ·Do not stream a schema against a non-native model. If you need tokens as they arrive, accept json_object and validate yourself, or move to a native model.

Related