Building a production RAG pipeline: architecture decisions that actually matter
Retrieval-Augmented Generation in a demo is twenty lines of Python. In production it is an operational system with five hard sub-problems. This is the architecture map we wish we had before shipping our first one.
Every RAG tutorial ends at the happy path: embed the docs, store the vectors, query at runtime, pipe to GPT, done. The demo works. Then you put it in front of real users and encounter latency spikes, hallucinations on edge-case queries, retrieval that surfaces the wrong chunk, and an embedding cost that doubles every quarter. This post is the architecture audit we run before shipping any RAG system.
The five hard sub-problems
- Chunking strategy — fixed-size splits lose sentence context; semantic splits lose position signals. The right answer depends on your document type.
- Embedding model choice — OpenAI text-embedding-3-large is not always the winner; domain-specific models outperform it on legal, medical and technical corpora.
- Retrieval quality — raw cosine similarity is baseline; hybrid search (BM25 + dense vector) adds 15–30% recall on most real corpora.
- Reranking — a cross-encoder reranker on the top-k candidates costs ~40ms and routinely doubles answer precision.
- Hallucination guardrails — citation grounding, faithfulness checks, and a deterministic fallback when retrieval confidence is low.
Chunking: the decision nobody talks about enough
The right chunk size is a function of your LLM's context window, your document structure, and your expected query type. For long technical PDFs, 512-token overlapping windows with 64-token overlap work well. For FAQ-style content, sentence-level chunking preserves answer units better. For code documentation, function-level chunking keeps signatures and docstrings together — splitting them is worse than not splitting at all.
# Sentence-aware chunking with overlap — production pattern
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=64,
separators=["\n\n", "\n", ". ", " ", ""],
length_function=len,
)
chunks = splitter.split_documents(raw_docs)
for i, chunk in enumerate(chunks):
chunk.metadata["chunk_index"] = i
chunk.metadata["source_hash"] = hash(chunk.page_content)Hybrid retrieval: why dense-only is not enough
Dense retrieval (vector similarity) excels at semantic matches — 'cheap hosting' finds 'affordable cloud servers'. Keyword retrieval (BM25) excels at exact matches — an error code, a product SKU, a person's name. Real user queries mix both. A hybrid retrieval pipeline runs both in parallel, fuses the ranked lists with Reciprocal Rank Fusion (RRF), and feeds the merged top-k to a reranker.
# Reciprocal Rank Fusion — merge dense + sparse results
def rrf_merge(dense_results, sparse_results, k=60):
scores = {}
for rank, doc in enumerate(dense_results):
doc_id = doc.metadata["source_hash"]
scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
for rank, doc in enumerate(sparse_results):
doc_id = doc.metadata["source_hash"]
scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
return sorted(scores.items(), key=lambda x: x[1], reverse=True)Reranking: the cheapest performance gain
A cross-encoder reranker (Cohere Rerank, bge-reranker, or a fine-tuned BERT model) scores each of the top-20 retrieved chunks against the full query — not just a vector distance, but actual relevance. The latency cost is 30–80ms. The precision gain is typically 15–40% on answer quality metrics. Ship this before you ship any prompt engineering — it is a better use of the same milliseconds.
Operational concerns: the bits the tutorials skip
- Index freshness — incremental upserts on document changes; track chunk hashes to skip unchanged content.
- Embedding cost — batch embed at ingest time; cache query embeddings with a 1-hour TTL for repeated questions.
- Latency budget — set a 2s p99 target; retrieval + rerank under 400ms leaving 1.6s for the LLM call.
- Evaluation — maintain a golden set of 50+ question/answer pairs; run recall@5 and faithfulness scores on every change.
- Fallback — route to deterministic lookup when retrieval confidence is below threshold.
The RAG demo is easy. The RAG product is an operational system: chunking, hybrid retrieval, reranking, evaluation, freshness, and a deterministic fallback. Build the eval harness first — everything else is a parameter you can tune against it.
Field notes authored by the core engineering and systems team at Chimbuk. We build and maintain production Flutter mobile applications, Next.js web architectures, and cloud backends. No recycled marketing theory — only architecture decisions, code patterns, and performance budgets from real production deployments.
Continue Reading
Related Engineering Guides
We architect and build the systems we write about.
From vector search pipelines to high-concurrency payment gateways — schedule a technical discovery call.