What RAG is

A language model is a machine that predicts the next word. Nothing more exotic than that. It was trained once, on a huge pile of text scraped before a cutoff date, and the training left behind a fixed set of weights. Those weights are frozen. When a question arrives, the model does not go and look anything up, because it has nothing to look things up in. It reads the words in front of it and produces the most plausible continuation.

That explains the failure exactly. "Our refund window is" has an overwhelmingly plausible continuation, learned from every refund policy on the public internet, and the model writes it with total confidence. It has no way to tell the difference between recalling and inventing, because internally both are the same operation. So the fix cannot be a better model or a sterner prompt. The fix has to be putting the actual text where the model can see it.

Which the model is happy to do. Give it your policy paragraph in the prompt and it will read the paragraph and answer from it. Everything else in RAG exists to work out which paragraph to give it.

The obvious move is to paste all your documents into the prompt on every call. That works fine for one handbook and falls apart the moment you have a real document set.

Size is the first wall you hit. Context windows are large now, though nowhere near large enough for a few thousand support tickets or a policy library that has been growing for a decade. Cost is the second, and it is relentless, because you pay per token of input on every single call, so a 200-page prompt answering "what is the refund window" bills you for 199 irrelevant pages every time any user asks anything. The third problem is the one people do not expect: the answers get worse. A model handed fifty pages loses the one line that mattered somewhere in the middle of them. Attention spreads across the input and the signal you cared about is diluted by everything sitting around it.

So a smaller prompt is not a compromise. Three well-chosen paragraphs beat fifty pages on answer quality while costing a fraction as much, which is the unusual case where the cheap option is also the better one.

When a question comes in, search your documents, take the handful of passages most relevant to that question, put only those in the prompt, and let the model write the answer from them. That is the whole idea. Retrieval is the search step. Generation is the model writing from what the search found.

The interesting part is what "most relevant" has to mean. Keyword search is not enough, because the person asking says "can I send this back" and the handbook says "returns are accepted within 30 days of delivery", and those two share almost no words. The search step has to match on meaning. That is the next page, and it is the piece that makes everything else possible.

Picture the store as a library where every paragraph has been filed by what it is about rather than by which book it came from. A question arrives. You do not read the library. You walk to the few paragraphs on the same subject as the question, photocopy them, and hand the photocopies to someone quick who writes an answer and marks which page each fact came from.

The model never reads your library. By the time it writes a word, someone has already pulled the three paragraphs that matter and handed it a photocopy. It's answering from a page it saw two seconds ago, not from anything it knows.

That is worth sitting with, because it reframes what you are actually building. You are not teaching a model your data. You are building a very good librarian and handing its output to a fluent writer who forgets everything between questions. When a RAG system gives a bad answer, the librarian is usually at fault, not the writer.

Doing this properly means running a parser for whatever files people upload, a vector index that finds the relevant passages at speed, a way to see why a given search returned what it did, and billing that ties the spend back to the right customer. Ringside runs those as one managed service behind an OpenAI-compatible API, so you make a few HTTP calls instead of operating a search stack.

Here is the whole flow, so the later pages have somewhere to hang.

The wire format is OpenAI's. If you have used their vector stores and the file_search tool, the calls here are the same shape, and the same SDKs work by pointing the base URL at Ringside. Where we do something different (open file types, graph retrieval, sealed stores, pricing you can read in two lines) the guide flags it as we go.