Inventory: what the assistant should know
- A list of sources
Network drives, SharePoint, systems, mailboxes with procedures. Where are the documents the assistant should answer from?
- An owner for each set
A person who knows what's current and is responsible for keeping it that way.
- What we leave out
For example HR data, contracts under special confidentiality or private correspondence.
- How often the documents change
That decides how often the assistant should refresh them.
Versions and duplicates
The most common cause of bad answers isn't a weak model but five versions of the same procedure. The assistant will find every one and may answer from the one from 2019.
A simple rule is enough to start: the assistant's folder holds only the version in force, with an “in force from” date in the name or in the text. The archive goes in a separate folder, outside the assistant.
Scans, tables and layout
- A scan is an image. Without text recognition (OCR) the assistant can't read it. Check the OCR quality on a few samples, especially tables and stamps.
- Tables are best left in a spreadsheet or described. A table cut in half when splitting into passages loses its meaning.
- Headings in documents are free help: the assistant splits text along them, so passages have coherent content.
- A two-column PDF is sometimes read in the wrong order. It's worth checking how it looks after conversion to text.
Passages and metadata
Before searching, documents are split into passages, usually a few hundred words each, ideally along the headings. Each passage gets metadata that lets the assistant filter and show the source.
- Title and source document
To show where the answer comes from.
- Date in force
To tell current from archived.
- Department or owner
To know whom to ask about a change.
- Access level
So the assistant shows a passage only to authorised people.
Permissions
The assistant must not be a back door. If someone has no access to the contracts folder, they must not get a passage of those contracts in an answer. Filtering by permissions happens at search time, before passages reach the model, not by asking the model to “not show” confidential things.
A set of test questions
Terms from this guide
- Chunk
A piece of a document, usually a few hundred words, into which texts are split before searching in RAG. Splitting along headings works best.
Questions and answers
Do all documents have to be tidied up before the start?
No. Start with one set, for example procedures or a department's knowledge base, check the quality on test questions and extend from there.
Can AI handle handwritten notes?
Poorly. Handwriting recognition is much less accurate than print. Such documents are better retyped or left out.
Who should keep the documents up to date?
The owner of each set. Without that, after a few months the assistant answers from outdated versions.
Sources and documentation
- Tesseract: OCR engine documentationtesseract-ocr.github.io
- Microsoft: chunking documents for searchlearn.microsoft.com
- Lewis et al. (2020): Retrieval-Augmented Generationarxiv.org