Where RAG goes wrong

Retrieval systems fail in a small number of specific ways, and almost all of them are invisible in a demo because a demo uses questions someone chose.

Here are the eight we see most. Use them as the questions to put to whoever is proposing to build yours, including us.

It answers confidently from a superseded document

What you see: The answer is well written, cited, and quotes the 2021 policy rather than the 2025 one. Both are in the system. Nothing flagged it.
Why it happens: Retrieval ranks on similarity to the question, and an old document about exactly the right topic scores higher than a new document that words things differently.
What fixes it: Version metadata at ingestion, and a retrieval filter that prefers current documents or refuses to answer from anything marked superseded. Also: actually archive the old ones. Most organisations have never deleted a policy in their history.

The rule and its exception end up in different chunks

What you see: The answer states the general rule and is technically quoting your document. It is also wrong for this case, because the "except where" clause was three hundred words further down.
Why it happens: Fixed-size chunking with no awareness of document structure. The split landed between the rule and the carve-out.
What fixes it: Chunk on structure — headings, clauses, numbered sections — with overlap between neighbours. Test specifically on the documents where exceptions matter, because that is where this bites.

It cannot find anything with a part number in it

What you see: Semantic questions work beautifully in the demo. Then someone searches "TX-4400-B" and gets nothing useful.
Why it happens: Vector search matches meaning, and an alphanumeric code has almost none. It is the single most common reason a system that demoed well is abandoned in month two.
What fixes it: Hybrid retrieval: run keyword search alongside vector search and merge the results. Not optional in any business with a parts catalogue, a clause numbering scheme or a project code.

It quietly shows people things they should not see

What you see: Nobody notices until they do, and then it is a serious conversation.
Why it happens: Documents were indexed without their access controls, so retrieval treats a restricted HR file the same as the staff handbook.
What fixes it: Carry permissions through ingestion and filter candidates by the asker’s entitlements before generation. Test it deliberately by having a low-privilege account try to extract something restricted.

Tables turn into nonsense

What you see: It reports a price or a tolerance from a table and the number belongs to a different row.
Why it happens: The PDF parser flattened the table into a stream of text and the row and column relationships were lost before anything else happened.
What fixes it: Structure-aware parsing for documents that are mostly tables, and in some cases extracting those tables to a separate queryable store rather than treating them as prose at all.

It answers when it should decline

What you see: A plausible answer to a question your documents genuinely do not cover.
Why it happens: The model was handed the closest passages available and instructed to answer. There is always a closest passage, even when nothing is relevant.
What fixes it: A relevance threshold below which the system says it does not know, and a prompt that makes declining an acceptable outcome. Expect to tune this — set it too tight and users get "I could not find that" for things you do have.

Quality degrades and nobody notices

What you see: It was good at launch. Six months on people have stopped using it and nobody logged a single complaint.
Why it happens: Documents were added, the corpus got noisier, retrieval got less precise. Without an evaluation set there was no signal, and users route around a tool rather than report it.
What fixes it: The evaluation set from day one, re-run on a schedule. Plus watching for the questions that returned nothing, which is the most useful log in the whole system.

It was built on documents that should have been fixed first

What you see: The system faithfully reflects that three departments have three contradictory versions of the same procedure.
Why it happens: Retrieval surfaces what exists. It does not adjudicate.
What fixes it: Not a technical fix. Somebody has to decide which document is authoritative. This is the most common reason we tell people to wait, and the advice is free.

The pattern in all of these

None of them are the language model being stupid. Seven of the eight are decisions made in ingestion and retrieval, before a model is involved at all, and the eighth is an organisational problem wearing a technical costume.

Which is why we spend most of a build on the plumbing and comparatively little on prompts. It is also why we sometimes say your documents are not ready, and mean it as advice rather than a negotiation.

What a build looks like