How RAG works in five steps
- Loading the documents
PDFs, text documents, wiki pages, emails. Text is extracted from each of them.
- Splitting into passages
Long documents are cut into pieces, for example paragraphs or sections, so that the search finds specific places rather than whole files.
- A meaning-based index
Each passage is turned into a vector that describes its meaning. Thanks to that, the question “how many days off do I have” finds the passage about “annual leave entitlement”.
- Searching at question time
The system finds the few passages that match the question best.
- An answer with sources
The model gets the question together with the passages and an instruction: answer only on their basis and cite the source. When passages are missing, it should say it doesn't know.
Why AI sometimes makes things up and how to limit it
A language model always wants to answer. When the search doesn't find the right passage, the model can put together a plausible-sounding answer from general knowledge. In a company that's dangerous, because the answer looks official.
- Require sources with every answer and show them to the user as links to documents.
- Give the model explicit permission to answer “I couldn't find this in the documents”.
- Limit the scope: an HR procedures assistant shouldn't answer questions about prices.
- Log questions without answers: that's a ready list of gaps in the documentation.
What most often hurts quality
| Problem | Symptom | Solution |
|---|---|---|
| Scans without a text layer | The system can't see the document's content | Text recognition (OCR) before loading |
| Several versions of the same document | Answers based on an old procedure | One source of truth, the archive outside the index |
| Tables and price lists | Rows and columns mixed up | Processing tables separately, sometimes converting them to text |
| Passages too large or too small | Answers without context or too general | Splitting by headings and sections, testing on questions |
| No permissions | An employee gets a passage they shouldn't see | Filtering results by the user's permissions |
| Jargon and company abbreviations | The search doesn't link an abbreviation to the full name | A glossary of abbreviations and hybrid search: meaning plus keywords |
Permissions are the most often overlooked topic. If an HR document ends up in a shared index, the assistant may quote from it to anyone. Permissions are set before the documents are loaded, not after.
How to check that the deployment works
The simplest and most effective method: collect 20 to 50 real questions employees ask every day, and for each write down the correct answer and the document it's in. After every change to the system, run all the questions and check how many answers are correct and have a source. Without such a set, every assessment is just an impression.
Terms from this guide
- RAG
Retrieval-Augmented Generation: the AI first searches for matching passages in documents and then answers based on them, citing the sources.
- Embedding (vector)
The meaning of a piece of text written down as numbers. It makes it possible to find passages with a similar meaning, even when they use different words.
- Hallucination
An AI answer that sounds plausible but is untrue or doesn't follow from the available data.
- OCR
Recognising text in images and scans so that a computer can search and process it.
Questions and answers
Does RAG require training the model?
No. The model stays as it is and gets the company knowledge as document passages at the moment of the question. That way, updating the knowledge means updating the documents, not retraining.
Can RAG run on your own server?
Yes. The document index, the search and the model can all run inside the company, without sending documents outside.
How many documents can be loaded?
Technically a great many. In practice it matters more that they're current and organised. It's better to start with one department and a well-working set of documents than with the whole company drive.