How to prepare a knowledge base for RAG: documents, chunking and accesses

RAG does not become useful after loading a PDF folder into a vector database. It begins to respond accurately only when the index contains relevant documents, understandable fragments, metadata and access rules. If these four parts are missing, the assistant either does not find the answer, or confidently quotes the old instructions, or shows the user data that he should not see.

Below is a practical procedure for preparing a knowledge base for RAG. It is suitable for internal assistant, support, sales department, CRM, Wiki and portal with regulations. If you need not only an index, but a ready-made working circuit with an interface, integrations and operation, this is included in the service creating an AI consultant with RAG.

What should be ready before uploading documents

Don't start by choosing a vector base or LLM. First, note four things:

1. Question scripts. Who will ask what decisions he makes based on the answer and what will be considered an error.
2. Base boundaries. Which sources are allowed for indexing and which cannot be included at all.
3. The owner of each source. A specific person or team responsible for keeping the document up to date.
4. Answer rules. When an assistant must cite a source, ask for clarification, or honestly answer “there is no data in the database.”

This protects the project from the typical pitfall of the team loading “everything they find” and then trying to cure the poor quality with a more expensive model. The model will not correct conflicting regulations, scans without text, and documents without an owner.

Step 1: Build a registry of sources, not an archive of files

Start with a table or directory of sources. It only contains eight fields:

Field	Example	Why is it necessary?
Source	Sales Wiki	Understand where the fragment came from
Owner	Sales Manager	Coordinate updates and controversial responses
Data type	Regulations, FAQ, contract, ticket	Select parsing and search rules
Users	Sales, support, managers	Set up access before searching
Refresh rate	Once a week	Schedule reindexing
Status	Current, under revision, archived	Do not mix old and current knowledge
Risk	Regular, confidential, PD	Limit indexing and retrieval
Link to original	Document URL	Show the user the verified source

First, take a small but high-quality set: for example, support FAQ, current regulations and product cards. A pilot is 50 good documents more useful than an index of 20,000 unsorted files.

Which sources usually give the best start?

• instructions that have one clear procedure;
• FAQ with confirmed answers;
• knowledge base articles and Wiki with the owner;
• letter templates, scripts and rules for processing applications;
• descriptions of products, tariffs and current conditions;
• closed tickets, but only after clearing personal data and highlighting standard solutions.

Be careful with correspondence, drafts, old presentations, and CRM exports. They often mix useful context with personal data, temporary arrangements and rules that have already been canceled.

Step 2: Clean and normalize documents

Text, structure, and relevance are important for searching, not the original file format. Before indexing, transform your documents into a predictable form: Markdown, HTML with headers, or structured JSON.

Check each source using a short checklist:

• PDF scans have high-quality OCR, and not just an image of the pages;
• Headings, lists and tables are read in the document;
• headers, footers, page numbers, repeating menus and service signatures have been removed;
• duplicates are merged or marked as versions of the same document;
• canceled documents are removed from the index;
• abbreviations and internal terms are spelled out at least once;
• a link to the original source remains next to the text.

Check the tables especially carefully. A table with tariffs, SLA or product parameters should not turn into a sequence of fragments like “30 days | 15% | 12,000.” It is better to store the table title, column headings and row as a single semantic block.

Step 3. Divide the document into meaningful fragments

A fragment, or chunk, is a unit that the search passes to the model. Poor partitioning leads to two problems: the desired idea ends up being cut in half, or one vector begins to describe five unrelated topics.

Start with document boundaries, not character counts:

1. First, divide by H1/H2, FAQ cards, regulations sections and table rows.
2. Within a large section, keep paragraphs and sentences, do not cut phrases in the middle.
3. Add the document title and heading path to each fragment. Then the excerpt from the middle of the file will not lose its context.
4. Apply slight overlap only where adjacent parts truly continue the same idea.

For normal text regulations, a reasonable starting hypothesis is about 512 tokens and 128 overlap tokens. This is not a universal norm: Microsoft recommends this as a starter option for vector search, but the optimal size depends on the document type and users' actual questions. Test multiple settings on your test set rather than blindly transferring settings.

When special chunking rules are needed

Content type	How to divide
FAQ	One question with a complete answer in one fragment
Regulations	Section or step of procedure, including conditions and exceptions
Table	Headers and one logical line or group of lines together
Agreement	Article or clause with number, title and associated definition
Ticket database	Problem, cause, solution and product version together
Catalog	One product or service card with characteristics and restrictions

Don't add the same introduction to all pieces for the sake of “better context”: this creates duplicates and makes searching worse. It is better to store shared information as metadata or a separate parent document.

Step 4. Add metadata that will be included in the search

The similarity vector answers the question “what is this fragment about?” Metadata answers the questions “which version is valid”, “which department is the document from”, “can it be shown to this person” and “where can I open the original”.

A minimal fragment card might look like this:

{
  "chunk_id": "support-refunds-v3-04",
  "document_id": "support-refunds-v3",
  "title": "Возвраты: правила и сроки",
  "section_path": "Поддержка > Возвраты > Деньги на карту",
  "source_url": "https://wiki.company.local/support/refunds",
  "updated_at": "2026-08-01",
  "valid_from": "2026-08-01",
  "owner": "head-of-support",
  "access_groups": ["support", "finance"],
  "confidentiality": "internal",
  "language": "ru",
  "content_type": "procedure",
  "version": 3
}

Required fields for most corporate projects:

• document_id and chunk_idto update and delete exactly the necessary data;
• title, section_path and source_urlto build quotes and the “where the answer is from” screen;
• updated_at, valid_from, version and status, so as not to present last year’s rule as valid;
• owner, so that a controversial answer can be sent to the owner of the knowledge;
• access_groups or another set of rights that filters the search before generating a response;
• language and content_type, if the database is multilingual or contains regulations, FAQs and cards.

Metadata does not need to be invented after indexing. These are best obtained from the original Wiki, CMS, CRM or source registry in the same pipeline where the document is cleaned and broken into fragments.

Step 5. Check access rights before generating a response

The most dangerous mistake in a corporate RAG is to first find a closed document, pass it to the model, and then try to cut out unnecessary things from the finished response. The restriction should work at the retrieval stage: the user should not receive either the fragment itself or its text for the model context in the search results.

Practical minimum:

1. Inherit ACLs or access groups from the source system.
2. Copy rights to each child fragment, not just the parent document.
3. For each search request, pass the allowed groups of the current user.
4. If you change permissions, reindex the affected document.
5. Test “access denied” scenarios separately, not just correct admin responses.

In the Azure AI Search documentation, this is called security trimming: results are trimmed by user or group IDs directly in the request. The specific implementation depends on the stack, but the principle is the same: searches control access, not prompts.

Step 6: Configure Update, Uninstall, and Versions

The knowledge base becomes outdated faster than it seems. Therefore, RAG does not need a one-time import, but a synchronization process.

For each document, store a hash of the content and the last processing time. Then the pipeline will be able to:

• do not recalculate embeddings for unchanged files;
• reindex only updated sections;
• delete fragments if the document is removed from publication;
• maintain a history of versions to analyze controversial answers;
• show the actual date in the assistant interface.

Do not leave old fragments in the index after replacing a document. Even if the new text is also loaded, the search may select the old version due to similar wording. If there are conflicting rules, it is useful to mark one document as active and the old one as archived or remove it from the production index completely.

Step 7. Test the database with real questions before launch

The quality of RAG should be measured in two stages: first the quality of the search, then the quality of the model response. A beautiful answer does not prove that the system got the right source.

Collect a test set of 50-100 questions from future users. Include:

• direct questions with one obvious answer;
• questions with official abbreviations and product names;
• questions where the answer is scattered across two documents;
• unanswered requests in the database;
• questions about new and repealed rules;
• questions from a user without rights to some documents.

For each question, state in advance the expected source and acceptable outcome. Then check:

Metrica	What does it show
Hit the source in top-k	Does retrieval find the required document before generation?
Response confirmation	Does the quoted passage provide a basis for each important statement?
Quality of failure	Does the assistant say “I don’t know” when there is no knowledge?
Relevance	Whether the current version of the document is selected
Access check	Is the closed fragment out of context or quotation?

If the search does not find the correct source, do not immediately rewrite the system prompt. More often the reason is the document structure, metadata, filters or fragment size.

Example workflow for two weeks

Days 1-2. We select one business scenario, collect questions, determine owners and access rights.

Days 3-5. We create a register of sources, clean up documents, set up OCR and normalization into a single format.

Days 6-8. We break the content into fragments, add metadata, build a primary index and links to the originals.

Days 9-10. We test retrieval on the control set, change the chunking and filtering rules.

Days 11-12. We enable the generation of responses with quotes, refusal rules and logging.

Days 13-14. We check access rights, launch the pilot on a limited group and record the database update process.

Such a pilot provides an answer to the main question before scaling: is the quality of the initial knowledge sufficient for users to really trust the assistant.

Common mistakes when preparing a knowledge base for RAG

1. The entire archive is indexed. It almost always contains duplicates, canceled rules and sensitive data.
2. The text is divided only by the number of characters. The question and answer, condition and exception appear in different fragments.
3. They do not store a link to the original. The user cannot check the answer, and the team does not understand what to fix.
4. Mix documents of different versions. The assistant receives several plausible but inconsistent answers.
5. They check permissions only in the UI. The closed fragment could already be in the model context.
6. Only successful requests are tested. Without “unanswered questions,” the system learns to hallucinate in a beautiful way.

Knowledge preparation usually takes longer than the first run of the model, but it determines whether RAG will save the team time or create new risks. If you need to link your knowledge base to a CRM, helpdesk, internal portal or multiple LLMs, see also how to launch a corporate RAG consultant and how to choose infrastructure for RAG.

FAQ

Is it possible to upload regular PDFs to RAG?

Yes, if the text is correctly extracted from the PDF and the structure is preserved. A scan without OCR, a complex table or presentation with text inside images require separate processing, otherwise the search will work on incomplete or distorted content.

What chunk size to choose for RAG

There is no one right size. For normal regulations, you can start with about 512 tokens and 128 overlap tokens, and then check the top-k sources on real questions. FAQs, tables, and legal clauses typically require more semantic breakdown.

Is metadata needed if there is a vector search?

Yes. Vector search finds semantic proximity, but without metadata it does not know the version, author, source, language, expiration date and access rights. Metadata allows you to filter documents and show the citation being verified.

Is it possible to give RAG access to all internal documents?

Not worth it. Only sources with a clear owner, relevance and access model are included in the index. It is better to exclude sensitive data or provide strict rights at the search level.

How often to update the knowledge base

It depends on the source: regulations can be synchronized by publication event, Wiki - according to a schedule, CRM or helpdesk data - incrementally. The main thing is that deleting and changing access rights are included in the index as reliably as adding new files.

Useful on the topic

• AI consultant with RAG for corporate knowledge base
• RAG consultant for a company: launching an AI knowledge base
• Server for RAG: on-premise, cloud and LLM selection
• AI2Media case: AI platform for SEO processes

Sources

• Microsoft Learn: document chunking for RAG and vector search
• Microsoft Learn: Snippet Cleanup and Metadata Enrichment
• Microsoft Learn: Document Access Control for Search
• Microsoft Learn: hybrid search for RAG
• OpenAI API: metadata and chunking strategy for vector store files
• OWASP GenAI: risks of retrieval augmentation and access control