How retrieval works, stage by stage
Six stages between a question and a cited answer. Each has one decision in it that determines whether the finished thing is trustworthy, and none of those decisions are visible in a demo.
Ingestion
Every document is pulled from wherever it lives — SharePoint, a file share, a DMS, an email archive — and converted to text. PDFs get parsed, scans get OCR, spreadsheets get flattened into something readable.
The decision that matters
How you handle documents that are mostly layout: drawings, forms, tables. A table read row-by-row instead of cell-by-cell produces confident nonsense later. This is where most quality is won or lost, and it is the least interesting stage, which is why it gets skipped.
Chunking
Documents are split into passages of a few hundred words. Retrieval works on passages, not whole files, because handing a model a 200-page manual defeats the purpose.
The decision that matters
Where you cut. Split mid-clause and a critical "except where" ends up in a different chunk from the rule it modifies. Splitting on headings and keeping a little overlap between neighbours costs storage and saves you from a whole category of wrong answer.
Embedding
Each passage is converted to a vector — a long list of numbers positioning it in meaning-space, so that passages about similar things sit near each other regardless of shared wording.
The decision that matters
Which embedding model, and whether it understands your vocabulary. A general model may not know that in your industry two terms mean the same thing. Testing this on your own documents before committing is half an hour that saves a rebuild.
Retrieval
The question is embedded the same way, and the closest passages are pulled back. In practice you want this hybrid: vector similarity for meaning, plus old-fashioned keyword search for exact terms.
The decision that matters
Pure vector search is bad at exact strings. Part numbers, clause references and product codes are precisely what people search for, and they are where semantic similarity is least useful. Hybrid is not a refinement, it is a requirement.
Reranking
A second, slower model re-scores the top candidates for actual relevance to the question, and the best few go forward.
The decision that matters
Whether to bother. It adds latency and cost. On a small tidy corpus you may not need it. On a large messy one it is often the difference between usable and not.
Generation and citation
The question and the surviving passages go to a language model with an instruction to answer from that material and cite it, and to say so when the material does not contain the answer.
The decision that matters
How hard you constrain it. Tight constraints mean more "I could not find this" and fewer inventions. That trade should be set by what a wrong answer costs you, which is very different for a marketing FAQ and a compliance lookup.
The part vendors skip: permissions
If your documents have access controls, the retrieval layer has to honour them. That means filtering candidate passages by what the person asking is allowed to see, before the model ever sees them.
Get it wrong and you have built a very efficient machine for leaking the salary review to the person it is about. It is not hard to do properly, but it has to be designed in at the retrieval stage. Bolting it on afterwards does not work, because by then the model has already read the passage.
Ask any vendor how permissions are enforced and at which stage. If the answer is vague, that is your answer.
How you tell whether it is any good
Build an evaluation set before you build the system: 50 to 100 real questions from real staff, each with a known correct answer and the document it lives in. Unglamorous, and it is the only way to know whether a change improved things or just moved the failures around.
Without it you are judging on vibes, and vibes are extremely generous to retrieval systems in the first fortnight.
Where this goes wrong