Skip to content
Back to blog
RAG

RAG Knowledge Systems: Architecture, Security, and Implementation

SFSufyan · AI EngineerMay 21, 202614 min read
RAG

Retrieval-augmented generation (RAG) is a pattern: search your own content, put the relevant passages into the model’s context, and ask the model to answer from those passages. The model is not “trained on your company” in the fine-tuning sense. It is shown a handful of excerpts at question time and asked to use them.

That distinction matters. RAG can quote a policy that changed last week, if the index was updated. It can also quote the wrong policy, an old version, or a document the user should not see. Quality is mostly a retrieval and permissions problem. This guide walks the architecture, the decisions that change answer quality, and the cases where you should not build a knowledge base at all.

Why a model needs external knowledge

A general model knows what was in its training data, up to a cutoff, and it will still produce a fluent answer when it does not know. Your price list, your exception process, yesterday’s contract redline, and the contents of a private ticket are not in that training set. Pasting them into a prompt every time does not scale, and putting them into a fine-tune makes updates slow and deletions unreliable.

External knowledge gives you a place to add, correct, and remove facts without retraining. The cost is a pipeline you have to operate.

Architecture

A production RAG system is a loop with a store beside it.

  1. Ingest sources and record their identity, owner, and access rule.
  2. Turn each source into passages (chunks) with metadata.
  3. Embed the passages and store the vectors plus the metadata.
  4. At question time, embed the question, filter by permission, and retrieve candidates.
  5. Optionally rerank the candidates.
  6. Build a context window from the winners, with citations attached.
  7. Generate an answer that stays inside that context, or says it does not know.

The stack guide covers where the database, the queue, and the model client sit. The rest of this article is the retrieval loop itself. Teams that want this built as a product usually start from our RAG knowledge systems work, often next to AI agents that can act only after the answer is grounded.

Ingestion and document processing

Ingestion is the unglamorous half. Connectors pull files, pages, tickets, or rows. Processing extracts text. Scanned PDFs need OCR. Slides and spreadsheets need a decision about what a “passage” is. Every document needs a stable ID, a source URL or path, a timestamp, and an access label copied from the source system, not invented later.

Keep the original file. The index is a derivative. When extraction is bad, you will want the original, not a guess about what the parser did. Record a content hash so an unchanged file is not re-embedded on every sync.

Chunking

Chunking splits a document into pieces small enough to retrieve and large enough to make sense. Too small, and you return a sentence with no condition attached. Too large, and you dilute the match and blow the context window.

Start with a structure-aware split: headings, list items, and paragraphs, with a modest overlap so a rule that begins at the end of one chunk is not orphaned. Do not use one chunk size for a policy manual and a chat log. Tables often need to stay intact or be summarized as rows with their headers repeated. There is no universal token count. Evaluate on your questions, then adjust.

Embeddings and the vector database

An embedding model maps text to a vector. Similar passages land near each other. The vector database returns the nearest stored vectors for a query vector. Use the same embedding model for documents and queries. If you change models, re-embed. Mixing two vector spaces in one index produces nonsense neighbors.

Store metadata beside the vector: document ID, tenant, access scope, updated time, title, and section heading. You will filter on those fields. A store that only keeps the vector forces you to retrieve first and authorize later, which is the wrong order.

Retrieval, filtering, and reranking

The first retrieval pass casts a wide net: nearest vectors that also match the filters. Filters are not optional decoration. Tenant, role, and document status (“published,” not “draft”) should be in the query. Keyword search alongside vectors helps when the user asks for an exact name, a SKU, or an error code that embeddings blur.

Reranking is a second model that scores the short list for actual relevance to the question. It costs latency and money, and it helps when the first pass returns neighbors that share vocabulary but not the answer. Add it when evaluation shows the right passage is in the top twenty but not in the top five. Do not add it to hide a broken chunking scheme.

Context construction and generation

Context construction decides what the model sees. Order passages so the most relevant come first, include the title and section so the model can name the source, and stop when you hit a token budget. Deduplicate chunks from the same paragraph. Drop passages below a score if your evaluation says low scores are usually noise.

The instruction to the model should be narrow: answer only from the passages, quote or cite the source, and say when the passages do not contain the answer. Then test that instruction with questions whose answer is absent. A system that always answers is a system that will invent.

Citations

A citation is a pointer the user can open: document title, section, and a link or ID that resolves in your app. Show it next to the claim, not in a footnote the interface hides. If you cannot attach a passage to a sentence, do not present the sentence as grounded. Citations are also how you debug. When an answer is wrong, you can see whether retrieval or generation failed.

Permission-aware RAG

Apply the user’s permissions as a filter before chunks are selected. The model should never receive text the user cannot open, because the model can repeat it. “Please don’t mention confidential files” is not a permission system.

This is harder than it sounds when permissions live in the source (a drive, a ticket tool, a CMS) and change. Sync the access labels on a schedule you can explain, and invalidate chunks when access is revoked. Test with two users in the same tenant who should see different sets. The security article on AI security and compliance covers the surrounding controls: logs, tool limits, and provider retention.

Data freshness

An index is stale the moment the source changes. Define a sync: webhooks where the source has them, polling where it does not, and a manual reindex for the rest. Delete or hide chunks when the source is deleted. Version the document so an answer can say which edition it used.

Freshness goals should match the domain. A staff handbook can lag by a day. A pricing answer that is wrong by a day may be unacceptable. If the fact changes by the minute, prefer a database query over a vector index.

Evaluation and hallucination reduction

Hallucination in RAG usually means one of three things: the right passage was not retrieved, the right passage was retrieved but buried, or the model added a claim that was not in the passages. Evaluate those separately.

  • Retrieval: for a question, is an acceptable passage in the top results?
  • Faithfulness: do the sentences in the answer follow from the cited passages?
  • Refusal: when the corpus does not contain the answer, does the system say so?

Build a small set from real questions, including unanswerable ones. Run it when you change chunking, embeddings, prompts, or models. A drop in retrieval with a prettier answer is a regression.

RAG, fine-tuning, and traditional search

Choose the mechanism that matches how the fact changes and how exact it must be.
ApproachGood atWeak at
Traditional search / SQLExact IDs, filters, known fieldsParaphrased questions over messy documents
RAGAnswers grounded in documents you can update and citeFacts that must be transactional and current to the second
Fine-tuningStyle, format, and narrow behavior you want by defaultKeeping specific, changing facts correct and deletable

Fine-tuning does not remove the need for retrieval when the knowledge changes or must be cited. Search does not go away either. Many good systems use keyword or SQL filters to choose a shelf, then vectors to choose a passage on that shelf.

Security and cost

Security is the permission filter, the tenant boundary, and the decision about whether prompts and chunks are retained by a model provider. Cost is embedding tokens on ingest, storage, query embeddings, rerankers, and generation tokens. Re-embedding an entire corpus because a prompt changed is wasted spend. Re-embedding because the embedding model changed is necessary. Track both.

A practical cost control is to index less. Not every file in a drive belongs in the corpus. Drafts, duplicates, and personal working folders add noise and risk. Curation is cheaper than a larger context window.

Common implementation mistakes

  • Dumping every file into one index with no owner and no access label.
  • Chunking only by token count and splitting tables and procedures in half.
  • Filtering permissions after the model has already seen the text.
  • Using a different embedding model for queries than for documents.
  • No evaluation set, so prompt tweaks are judged by a single demo question.
  • No deletion path, so removed documents keep answering.
  • Asking the model questions that a database already answers exactly.

When RAG is the wrong solution

Skip RAG when the source of truth is a small set of structured fields, when the user needs a transaction rather than an explanation, or when you cannot legally copy the documents into an index you operate. Skip it when you have fewer than a handful of pages, put those pages in the prompt or in the product UI. Skip it when nobody will maintain the corpus. A stale index is worse than a link to the handbook, because it sounds current.

Implementation checklist

  • Name the sources, the owners, and the access model.
  • Store originals, IDs, timestamps, and content hashes.
  • Chunk in a way that keeps a rule or a row understandable.
  • Embed with one model, and plan a re-embed path.
  • Filter by tenant and permission inside the query.
  • Cite a passage the user can open.
  • Refuse when the passages do not contain the answer.
  • Sync and delete on a stated schedule.
  • Keep a question set for retrieval, faithfulness, and refusal.
  • Log document IDs and model metadata under your retention rules.

Conclusion

A RAG system is a search product that happens to speak in paragraphs. Treat ingestion, permissions, freshness, and evaluation as the product. The model will only be as careful as the passages you hand it. If those passages are the wrong shelf, workflow automation or a direct integration will serve the user better than a more elaborate prompt.

Frequently asked questions

Is RAG the same as training a model on our data?

No. RAG retrieves passages at question time and puts them in the prompt. The base model’s weights stay as the provider shipped them, unless you separately fine-tune. Updating a document means updating the index, not running a training job.

How many documents do we need before RAG is worth it?

Enough that people cannot reliably remember where the answer lives, and enough change that copying text into a prompt is brittle. A five-page handbook can live in the product as a page. A changing library of policies, tickets, or specs is where retrieval starts to pay for its pipeline.

Can RAG stop hallucinations entirely?

It can reduce unsupported answers if retrieval finds the right passage and the product is willing to say “not in the sources.” It cannot stop a model from embellishing if you reward it for always producing an answer. Measure faithfulness and refusal, and treat misses as bugs.

SF

Sufyan

AI Engineer at Vyntrix Labs

Let's build something intelligent

Ready to automate the busywork and scale with AI?

Book a free discovery call and we'll map your highest-impact automation opportunities, no obligation, no jargon.