Ask a general-purpose model a question about your own business and you get one of two bad outcomes. Either it politely tells you it has no idea what your refund policy is, or — the one that actually causes incidents — it makes up an answer that reads perfectly and is completely wrong. I watched a support assistant once confidently quote a 60-day return window for a company whose real policy was 30. Nobody had told it the real number. It just filled the gap with something plausible, because that is what these models do when they don't know: they generate.
Retrieval-Augmented Generation is the fix for that gap, and it has quietly become the default way teams build enterprise AI assistants. The idea is simple enough to explain in a sentence. The engineering is where it gets interesting, and where most of the difference between a demo and a dependable system lives.
What RAG actually does
A base model knows only what it saw during training. That knowledge is frozen at a point in time and contains nothing proprietary to you — not your product catalogue, not last week's policy update, not the ticket history your support team relies on. RAG stops treating the model as a knowledge store and starts treating it as a reasoning engine you feed facts to.
At the moment a question is asked, RAG retrieves the most relevant passages from your own sources and drops them into the prompt alongside the question. The model then answers from that material rather than from memory. Because the answer is built from specific retrieved passages, you can show the user exactly where each claim came from. That citation is not a nice-to-have. In a regulated business it is often the whole reason the project is allowed to ship.
How the pipeline works
There are two halves to a RAG system, and people usually only picture the second one. The first half happens before anyone asks anything.
INDEXING (offline, ahead of time)
your documents
|
v
split into chunks --> embed each chunk --> store vectors + text
in a vector DB
RETRIEVAL + GENERATION (at question time)
user question
|
v
embed the question
|
v
find nearest chunks in the vector DB <-- the make-or-break step
|
v
assemble prompt: system instructions + retrieved chunks + question
|
v
LLM generates a grounded answer, with citations back to the chunks
The offline half turns your corpus into something searchable by meaning. You break documents into chunks, convert each chunk into an embedding — a long list of numbers that captures what the text is about — and store those in a vector database. The online half embeds the incoming question the same way and looks for the chunks whose vectors sit closest to it. Closeness in that vector space roughly means closeness in meaning, so a question about “getting my money back” can match a passage titled “Refunds” even though they share no words.
The step I've marked as make-or-break is retrieval. Everything downstream depends on it. If the right chunk doesn't come back, the model has nothing to ground itself in, and no amount of clever prompting saves you. A surprising number of “the AI gave a bad answer” complaints are really “retrieval returned the wrong passage” in disguise.
Why teams choose RAG over fine-tuning
This is the question I get asked most, usually phrased as “can't we just train the model on our data?” You can fine-tune, but for factual knowledge it's usually the wrong tool, and it helps to be clear about why.
Fine-tuning bakes new behaviour into the model's weights. It's excellent for teaching a model a style, a format, or a task — respond like our brand voice, always output valid JSON, classify tickets into these categories. It is a poor fit for facts that change. Your policies, prices, and inventory move constantly, and every change would mean assembling a dataset and running another training job. By the time it's done, something else has changed.
RAG keeps the knowledge outside the model, in a store you can edit like any other database. Add a document this morning and the assistant can answer questions about it this afternoon — no retraining, no redeploy. And because the answer traces back to a specific source, it's auditable in a way a fine-tuned model's recall never is. When someone asks “why did it say that?”, RAG can point at the paragraph. Fine-tuning can only shrug.
In practice the two aren't rivals. A mature system often fine-tunes for behaviour and uses RAG for knowledge. But if your problem is “the model doesn't know our stuff,” reach for RAG first.
What makes a RAG system good
The gap between a RAG demo and a RAG product is almost entirely in the unglamorous details, and they're worth naming because they're where the effort actually goes.
Chunking sounds trivial and isn't. Chunk too large and you dilute the relevant sentence in a wall of surrounding text, which drags down retrieval accuracy and wastes tokens. Chunk too small and you sever a fact from the context that gives it meaning. The classic mistake I see is splitting purely by character count, which happily cuts a table off from its header or a clause off from the sentence that qualifies it. Chunking that respects document structure — sections, headings, table boundaries — beats naive splitting almost every time.
Embedding quality sets your ceiling. A weak embedding model puts unrelated passages close together and separates related ones, and retrieval inherits that confusion. Tables and images are their own headache; a chart carries information that plain text extraction throws away, so you either describe it in text during indexing or accept that the assistant is blind to it.
Then there's the prompt. The instruction to the model has to do two jobs: answer only from the retrieved context, and say plainly when the context doesn't contain the answer. That second instruction is the one teams forget, and it's what stands between “I don't have that information” and a confident fabrication. Getting a model to admit ignorance is genuinely harder than getting it to answer, and it takes deliberate prompting plus testing.
None of this is guesswork if you evaluate. Build a set of real questions with known-good answers and score retrieval and generation against it every time you change something. Without that, you're tuning chunk sizes by feel and hoping.
Common pitfalls
The failure modes repeat across projects, and every one of them is avoidable.
Retrieving the wrong amount of context. Too little and the model is missing the piece it needs. Too much and the relevant passage gets buried among marginally related ones — models genuinely lose the thread when you stuff the context full, and answer quality drops even though you “gave it more to work with.” More context is not more accuracy.
Ignoring document structure. Flattening everything into undifferentiated text throws away signal that headings, tables, and hierarchy were carrying. The retriever can't recover what the ingestion step destroyed.
Letting the index drift. This is the quiet killer. A source document gets updated but the vector store still holds the old chunk, so the assistant serves last quarter's policy with total confidence. Your re-indexing pipeline matters as much as your retrieval logic, and it's the part that's easy to skip because nothing errors when you do — the answers just slowly go stale.
Skipping evaluation. Teams that don't measure ship changes that fix one case and break three, and they find out from users instead of from a test run. An eval set built early is the cheapest insurance in the whole project.
Key takeaways
RAG lets a general model answer accurately about your specific business, cite its sources, and stay current — all without the cost and rigidity of retraining. It works by retrieving relevant passages at question time and grounding the answer in them, which is why the retrieval step, not the model, is where you should spend your attention.
Treat the model as a reasoning engine and your content as the source of truth. Chunk with respect for structure, embed well, keep the index fresh, tell the model to admit when it doesn't know, and evaluate on real questions. Do those, and you get an assistant people can actually trust — which is the only kind worth building.
Interested in a RAG assistant grounded in your own knowledge base? See our Generative AI & LLM Solutions.
