Secure RAG for regulated data

What this gives you and why it exists

Standard RAG stores your documents in a vector database in the clear. Anyone with a database dump, a backup or host access can read the chunks. For most workloads that is fine. For regulated data (patient records, KYC files, privileged legal text, source you are contractually barred from exposing) it is not, because the threat model includes us, the operator. A bring-your-own-key (byok) vector store removes Ringside from that threat model: the data encryption key (DEK) that seals every chunk and vector is wrapped under a passphrase you hold and we never persist.

Mechanically, each sealed store has a random per-store 32-byte DEK. Chunk text is sealed with AES-256-GCM under that DEK, and with vector_sealing: full the embedding vectors are sealed too, so nothing about the corpus leaks at rest. The difference between custody modes is who can unwrap the DEK. Under managed, the DEK is wrapped with our server key and we can always open it. Under byok, the DEK is wrapped under a key derived (scrypt) from your passphrase, and we store only the wrap, a salt and a verifier. With no active key lease, a byok store is opaque to us. That is the entire point.

The lifecycle

create (encryption=byok, key passphrase)
        |
        v
   [ UNLOCKED ]  <-- first lease opened automatically at create
        |   ^
 ingest |   | POST .../unlock   (or FC-Vector-Store-Key header)
 query  |   |
        v   |
   roll TTL on every cryptographic use (1 h idle window)
        |
        | 1 h with no ingest/query   OR   POST .../lock
        v
   [ LOCKED ]  --> ingest fails the file with vault_locked;
                   retrieval returns no results

1. 01Create the store with encryption: "byok", vector_sealing: "full" and a passphrase key. Create opens the first key lease for you, so you can ingest right away.
2. 02Attach files. Ingest is async; poll each file row until status is completed.
3. 03Query via the file_search tool inside an Assistants run while a lease is active.
4. 04Lock the store when you are done, or let the 1-hour idle window expire.
5. 05Next time, present the passphrase via POST .../unlock (or the FC-Vector-Store-Key header) to take a fresh lease.

Create a sealed byok store

Send a POST to /v1/vector_stores with the custody fields and a passphrase key. The passphrase must be at least 12 characters. It is used once to derive the wrapping key and is never stored.

curl https://api.fightclub.pro/v1/vector_stores \
  -H "Authorization: Bearer $FC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "kyc-case-files",
    "embedding_model": "text-embedding-3-small",
    "encryption": "byok",
    "vector_sealing": "full",
    "key": { "type": "passphrase", "passphrase": "correct-horse-battery-staple" }
  }'

Create-time parameters that matter for sealing

encryption

none | managed | byok, default none. byok is the only mode where we cannot read the store at rest. A bad value returns 400 invalid_encryption.

vector_sealing

none | source | full, defaults to full when the store is sealed and is ignored when encryption is none. source seals the original chunk text only; full also seals the embedding vectors. Setting it on a plaintext store returns 400 invalid_vector_sealing.

key

Required when encryption is byok. Shape { "type": "passphrase", "passphrase": "<>=12 chars>" }. type: "kms" is reserved for cloud-KMS custody and currently returns 400 byok_kms_unavailable. A missing or malformed key returns 400 byok_key_required; a short passphrase returns 400 byok_passphrase_too_short.

vector_index

auto | flat | ivf, default auto. ivf is for large sealed corpora (see below) and requires a sealed store; requesting it on a plaintext store returns 400 ivf_requires_encryption.

embedding_model

Fixed at create. Change it later only by running a migration (POST .../migrate), which re-embeds and swaps the index atomically.

Take a key lease

A key lease is a short-lived, server-key-encrypted cache of the unwrapped DEK, stored exactly like a vault session. Create opens the first lease automatically. After that, every byok store starts each session locked, and ingest or query needs an active lease. You open one in two ways.

Explicit lease: POST .../unlock

curl https://api.fightclub.pro/v1/vector_stores/vs_8fK2pQ7r/unlock \
  -H "Authorization: Bearer $FC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "key": { "passphrase": "correct-horse-battery-staple" } }'

The body accepts either { "key": { "passphrase": "..." } } or a bare { "passphrase": "..." }. A wrong passphrase returns 401 invalid_passphrase (we cannot tell you whether it was close; the verifier either matches or it does not). Calling /unlock on a non-byok store returns 400 not_a_byok_store, and an empty passphrase returns 400 missing_passphrase.

Per-request lease: the FC-Vector-Store-Key header

For a one-shot ingest or query you can skip the separate /unlock round trip and present the passphrase inline with the FC-Vector-Store-Key request header. It takes a lease for the duration of the call. The passphrase is still never persisted. Use the explicit /unlock endpoint when you are about to do a burst of ingest or many queries, so you authenticate the passphrase once and ride the rolling window.

curl https://api.fightclub.pro/v1/vector_stores/vs_8fK2pQ7r/files \
  -H "Authorization: Bearer $FC_API_KEY" \
  -H "FC-Vector-Store-Key: correct-horse-battery-staple" \
  -H "Content-Type: application/json" \
  -d '{ "file_id": "file_3Jq9Zt" }'

Ingest under a lease

Upload your documents as files first (purpose does not matter for attach), then attach them to the store. Ingest is asynchronous. Attach returns immediately with the file row in a pending state; the rag-worker fetches a transient lease, parses, chunks, embeds and seals each chunk under the DEK. Poll the file row until status is completed.

STORE=vs_8fK2pQ7r
KEY=correct-horse-battery-staple

# Open a lease for the whole ingest burst
curl -s https://api.fightclub.pro/v1/vector_stores/$STORE/unlock \
  -H "Authorization: Bearer $FC_API_KEY" -H "Content-Type: application/json" \
  -d "{\"key\":{\"passphrase\":\"$KEY\"}}" >/dev/null

# Attach a file (lease already open, no per-call key needed)
FILE=$(curl -s https://api.fightclub.pro/v1/vector_stores/$STORE/files \
  -H "Authorization: Bearer $FC_API_KEY" -H "Content-Type: application/json" \
  -d '{"file_id":"file_3Jq9Zt"}' | jq -r .id)

# Poll until completed
while :; do
  S=$(curl -s https://api.fightclub.pro/v1/vector_stores/$STORE/files/$FILE \
        -H "Authorization: Bearer $FC_API_KEY" | jq -r .status)
  echo "status=$S"; [ "$S" = completed ] || [ "$S" = failed ] && break
  sleep 2
done

For bulk loads, attach up to 500 files in one call with POST .../file_batches. A larger batch returns 400 too_many_files. If the lease expires mid-ingest because the corpus is large and slow, individual files fail closed rather than ingesting in the clear; take a fresh lease and retry the failed rows.

Query under a lease

There is no direct query endpoint. You retrieve from a store through the OpenAI-compatible file_search tool inside an Assistants run, which keeps the same surface for sealed and plaintext stores. Attach the store id under tool_resources.file_search.vector_store_ids and hold a lease open (or present FC-Vector-Store-Key) for the duration of the run. The DEK is unwrapped in the web tier to open chunks and rank vectors, then dropped.

curl https://api.fightclub.pro/v1/assistants \
  -H "Authorization: Bearer $FC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "fc:openai/gpt-4o-mini",
    "instructions": "Answer only from the attached case files. Cite the source.",
    "tools": [{ "type": "file_search" }],
    "tool_resources": { "file_search": { "vector_store_ids": ["vs_8fK2pQ7r"] } }
  }'

If the store has GraphRAG on, file_search also returns a facts array of {subject, predicate, object} relationships next to the chunk citations. On a sealed store the labels and predicates are sealed under the store DEK and resolved in-tier, so the graph stays encrypted at rest while topology stays usable. GraphRAG bills as its own per-GB-day storage line, with the first 1 GB-day per store per day free.

Lock when idle

When a working session ends, drop the lease explicitly rather than waiting for the idle timer. Locking shrinks the window in which a compromised web tier could read the DEK from the unlock-session cache.

curl https://api.fightclub.pro/v1/vector_stores/vs_8fK2pQ7r/lock \
  -H "Authorization: Bearer $FC_API_KEY"

After locking, the store fails closed. A new ingest under no lease marks the file failed with last_error.code set to vault_locked. A retrieval run against a locked store returns no chunks rather than surfacing ciphertext, so a run that comes back empty after an idle gap is your signal to re-/unlock (or pass FC-Vector-Store-Key) and retry. Branch on the code, never the human message.

Large sealed corpora: the ivf index

Sealing has a cost. With vector_sealing: full, the vectors are ciphertext at rest, so the default warm path (vector_index: auto, which resolves to flat on small stores) must decrypt vectors to rank them and holds the working set in RAM. That is fast and leaks nothing, but it does not scale to very large stores you would rather not hold decrypted in memory. For those, set vector_index: "ivf".

Errors and edge cases

A locked byok store does not reject ingest or query synchronously. There is no 409 here. Attach still returns 201; the async ingest then fails the file row, setting status to failed and last_error.code to vault_locked, because the worker could not get the DEK without a lease. A retrieval run against a locked store returns no chunks. So a lock surfaces as a failed file or an empty run, never as ciphertext.

Every error body follows the standard Ringside shape: { "error": { "type", "code", "message", "param"? } }. The request id is in the X-Request-Id response header, never the body. Always branch on code. A wrong-passphrase invalid_passphrase at /unlock and a vault_locked file failure call for different recovery (fix the secret versus take a fresh lease and retry the file), so do not pattern-match on the message string.

Limits and defaults to plan around

• ·Passphrase: minimum 12 characters, derived once, never stored.
• ·Lease TTL: 1 hour idle, rolled forward on every cryptographic use.
• ·File batch: up to 500 files per file_batches call (too_many_files past that).
• ·Store metadata: 4 KiB serialized, enforced on create and update (metadata_too_large past that). Keep to the platform metadata convention of at most 16 keys, key <=64 chars and value <=512 chars.
• ·File upload size: 512 MB per file.
• ·Query log pagination: GET .../queries defaults to 50, max 200; other lists default 20, max 100.
• ·Embedding model is fixed at create; change it only via POST .../migrate.

A safe operating pattern

1. 01Store the passphrase in your own secrets manager (Vault, a KMS-backed secret, an HSM-fronted store). Never in code, env files you commit or logs.
2. 02Open one lease per work session with /unlock, do all ingest and queries inside the rolling window, then /lock explicitly when done.
3. 03Prefer FC-Vector-Store-Key for genuinely one-shot calls so no lease lingers; prefer /unlock plus /lock for bursts.
4. 04Default to vector_sealing: full and vector_index: auto. Only reach for ivf once the corpus is large enough that warm scanning hurts, and accept the coarse-topology leak when you do.
5. 05Treat a vault_locked file failure or an empty retrieval run as a re-lease-and-retry signal, not an error to surface to end users.