4 min read
What Drives the Cost of a RAG System? Build and Running Costs Explained
A practical breakdown of what makes a RAG project cheap or expensive to build and to run: data complexity, integrations, access control, evaluation, embeddings, vector storage and per-query token costs, with the math to estimate your own.
Enterprise AI/ML Engineer
- RAG
- Cost
- Architecture
- Enterprise AI
"How much does a RAG system cost?" is the most common question I get, and the honest answer is that two projects with the same one-line description can differ by a factor of five. The difference is almost never the model. It's the data, the integrations and the guarantees the system has to give.
This guide breaks the cost into its real drivers so you can reason about your own project, and shows the simple math behind running costs. If you want a tailored timeline instead, the AI project estimator builds one in a minute.
Two different bills: building and running
A RAG system has two kinds of cost that behave very differently:
- Build cost is engineering time: understanding the data, building ingestion and retrieval, integrating with your systems, evaluating and hardening. It's mostly fixed and front-loaded.
- Running cost is infrastructure: embedding documents, storing vectors, serving queries through a language model, hosting and monitoring. It scales with data volume and usage.
Most budgets underestimate the first and overestimate the second.
What drives build cost
1. How messy the documents are
Clean, born-digital documents in one format are the cheap case. Costs rise with scanned files that need OCR, tables and forms whose structure carries meaning, mixed languages, and inconsistent formatting. Ingestion is where most RAG projects quietly spend their time, because retrieval can only be as good as the text it searches. (See the OCR-to-RAG pipeline for what that work looks like.)
2. How many sources and integrations
One folder of PDFs is a different project from a system that syncs SharePoint, a CRM, a ticketing tool and a database. Each source needs connectors, change detection, error handling and its own permissions model.
3. Who is allowed to see what
If every user can see every document, retrieval is simple. If access depends on role, team, client or region, the permissions have to be enforced inside retrieval, before ranking, and tested. This is non-negotiable for legal, healthcare, finance and public-sector work, and it is real engineering.
4. Personal data and compliance
When documents contain personal data, anonymization belongs in the pipeline before anything is indexed. Building it well means detection across several methods, consistent masking, stricter rules for sensitive groups and measured accuracy. On a legal project this reached 98% detection, and it was a project in itself (the anonymization engine).
5. How good it has to be, and how you'll know
A demo that looks right on ten questions is cheap. A system you can trust needs an evaluation set built with domain experts, automated scoring and several iterations of chunking, hybrid search and re-ranking until the numbers hold. This is the line item most often cut, and the one whose absence most often sinks projects.
6. Scale and latency targets
Serving a few internal users is different from 10,000+ queries a day or a hard requirement to answer in under two seconds. Throughput, caching and model choice all become design work.
What drives running cost
Running costs are easier to estimate because they follow simple arithmetic. Prices change often, so the useful skill is knowing the formula and plugging in your provider's current rates.
Embedding the corpus (mostly one-time)
Total tokens to embed is roughly documents × pages per document × tokens per page. For example, 50,000 documents × 10 pages × 500 tokens per page is about 250 million tokens. You pay this once, plus incremental costs as documents change, and again only if you switch embedding models.
Storing vectors
Each chunk becomes one vector. With 512-token chunks, those 250 million tokens produce roughly 490,000 chunks. A 1,536-dimension vector stored as 32-bit floats takes about 6 KB, so the raw vectors come to around 3 GB, plus metadata and index overhead. That's modest for any managed vector database. The storage tier you choose matters more than the raw size.
Answering questions (the recurring cost)
Each query sends tokens to the language model:
| Component | Typical size |
|---|---|
| System instructions | 1,000–2,000 tokens |
| Retrieved context (5 chunks × 512) | about 2,500 tokens |
| Conversation history | 0–2,000 tokens |
| Generated answer | 200–500 tokens |
Multiply the input and output tokens by your model's price per million tokens and by your expected query volume. Two levers dominate:
- Prompt caching. The system instructions are identical on every request; caching them cut input-token costs by about 90% on one production SaaS backend.
- Model routing. Simple questions don't need the largest model. Routing them to a smaller one often halves the bill with no visible quality loss.
Better retrieval also lowers cost: five precise chunks beat fifteen vague ones, for both quality and tokens.
Where projects overspend
- Over-engineering the first release. Start with one high-value use case, prove it on real data, then expand.
- Skipping evaluation. Teams without an evaluation set iterate by intuition and pay for it in rework.
- Oversized context windows. Stuffing more text into the prompt is the most expensive way to compensate for weak retrieval.
- Ignoring access control until late. Retrofitting permissions into retrieval is far more expensive than designing them in.
A sensible way to buy a RAG system
Treat it in phases. A short discovery and proof of value on your real documents answers the expensive unknowns: data quality, achievable accuracy and integration effort. A build phase then delivers the production system in milestones with weekly demos, followed by hardening: evaluation, guardrails, monitoring and handover.
That structure is how I run enterprise RAG engagements, and why the estimate after a discovery call can be a fixed quote rather than a guess. To see what your project's phases might look like, try the project estimator or book a free discovery call.