5 min read
Enterprise RAG Architecture: A Production Checklist
The decisions that separate a RAG demo from a production system: evaluation, ingestion, hybrid retrieval, access control, PII, citations, cost and observability, as a checklist you can use on your own project.
Enterprise AI/ML Engineer
- RAG
- Enterprise AI
- Architecture
- Evaluation
Retrieval-Augmented Generation looks simple on a whiteboard: embed your documents, retrieve the closest chunks, hand them to an LLM. A weekend prototype over a handful of PDFs usually works. Then the real corpus arrives (tens of thousands of documents, scanned files, tables, three languages, strict access rules) and answer quality collapses.
This checklist is what I work through on every enterprise RAG engagement, including a 50,000-document legal platform that runs in production. Use it to pressure-test your own design.
1. Build the evaluation set first
Before choosing a chunk size, a vector database or a model, write down 100–200 real questions with the answers and source passages a domain expert would accept. Every later decision gets scored against this set.
Without it, teams "improve" retrieval by intuition and ship regressions they never notice. With it, you can say that a change moved retrieval hit-rate from 81% to 88% and faithfulness held steady, which is a sentence stakeholders trust.
Checklist:
- Questions written with real users or subject-matter experts
- Expected source documents recorded for each question
- Automated scoring for retrieval hit-rate, faithfulness and answer relevance (RAGAS or similar)
- Re-run on every pipeline change, not just before launch
2. Treat ingestion as a product
Most RAG failures are ingestion failures. If the text extracted from a PDF is scrambled, no model can recover it.
- Parse by format. Native PDFs, Word files, HTML and scanned images each need their own path. Scanned archives need OCR with image preprocessing, or they become noise in your index. (See the OCR-to-RAG pipeline for one approach.)
- Preserve structure. Headings, section numbers, tables and lists carry meaning. Keep them as metadata or in the chunk text.
- Make it repeatable. Ingestion should be a pipeline you can re-run when documents change, not a one-off notebook.
3. Choose chunking for your documents, not by default
There is no universal chunk size. Dense legal paragraphs, short FAQ entries and long technical manuals behave differently. On the legal platform, 512-token chunks with overlap beat both 256 and 1,024 tokens, but only the evaluation set could prove it.
- Split on structure (sections, clauses) before falling back to token counts
- Use overlap so reasoning isn't cut mid-argument
- Store the parent section so you can expand context at answer time
4. Use hybrid retrieval and re-ranking
Pure vector search misses exact identifiers: case numbers, product codes, legal citations, names. Pure keyword search misses meaning. Combining BM25 with dense vectors is the single most reliable upgrade I know of; on the legal platform it improved retrieval accuracy by 15% over either method alone.
- Hybrid BM25 + vector retrieval, fused with reciprocal rank fusion or weighted scores
- A re-ranker (Cohere Rerank or a cross-encoder) over the top 20–50 candidates
- Query rewriting or routing for questions that need a specific index or time period
5. Make metadata a first-class citizen
Filters often matter more than embeddings. "Decisions from the appeals court in 2023" is a metadata query with a semantic component, not a semantic query.
- Index dates, document types, jurisdictions, departments and owners
- Let the query router extract filters from natural language
- Use dedicated indices for distinct access patterns when it simplifies retrieval
6. Enforce access control at retrieval time
If a user can't open a document, the assistant must not quote it. Hiding the source citation doesn't help, because the content still leaks through the answer.
- Permission-aware retrieval: filter by the user's roles before ranking
- Namespace isolation for distinct audiences or tenants (one public-sector assistant serves the public and field teams from one knowledge base, with namespaces keeping their content apart)
- Audit logs of who asked what and which sources were returned
7. Handle personal data before indexing
In regulated domains, anonymization belongs in the ingestion pipeline, not the prompt. Once PII is in your vector store, it's in every downstream answer and backup.
- Detect and mask names, addresses, identifiers and financial data before embedding
- Add stricter rules for sensitive groups such as minors
- Measure detection accuracy against a labeled sample; 98% was the bar on the legal project
8. Ground every answer and cite it
- Instruct the model to answer only from the retrieved context and to say so when evidence is missing
- Return citations users can click through to verify
- Use structured outputs so the UI can render sources, confidence and follow-ups reliably
9. Engineer for latency and cost
- Cache stable prompt prefixes; prompt caching cut input-token costs by about 90% on one production SaaS backend
- Route simple questions to smaller, cheaper models
- Keep the context window tight: better retrieval beats bigger prompts
- Set a latency budget (for example, under 2 seconds end to end) and measure p95, not averages
10. Observe everything after launch
- Trace each request: query, rewritten query, retrieved chunks, scores, prompt, answer, latency, cost
- Capture user feedback (thumbs, corrections) and feed bad answers back into the evaluation set
- Alert on retrieval drift when new documents change answer quality
The checklist at a glance
| Area | Production-ready means |
|---|---|
| Evaluation | Expert-built question set, automated scoring on every change |
| Ingestion | Format-specific parsing, OCR for scans, repeatable pipeline |
| Chunking | Structure-aware, overlap, tuned against the eval set |
| Retrieval | Hybrid BM25 + vectors, re-ranking, query routing |
| Metadata | Filters for dates, types, jurisdictions, owners |
| Access | Permission-aware retrieval, namespaces, audit logs |
| Privacy | PII anonymized before indexing, accuracy measured |
| Answers | Grounded, cited, structured outputs |
| Cost & speed | Prompt caching, model routing, latency budget |
| Operations | Tracing, feedback loop, drift alerts |
Where to start
If your RAG pilot is stuck between "impressive demo" and "trusted tool", the gap is almost always in the first three items: evaluation, ingestion and chunking. Fix those and the rest becomes measurable.
I design and build these systems end to end. See my enterprise RAG development services, or book a free discovery call to talk through your corpus and constraints.