RAG Architecture Explained: Components, Patterns, and Production Considerations

Five architectural decisions determine whether a RAG system works in production. I’ve watched teams make all five correctly and ship something people trust with questions that matter. I’ve watched teams make two of them wrong and spend six months debugging a pipeline that was architecturally broken from week one. Wrong decisions at the retrieval, embedding, or context-assembly layer compound invisibly until production traffic exposes them.

This post assumes you already know what RAG is and why you want one. It’s for the engineer, tech lead, or AI architect who has to decide how to build it: which embedding model, which chunking boundary, which retrieval pattern, which LLM, and how to evaluate the whole thing before it goes live.

Decision 1: Which embedding model

What you’re actually choosing between

The embedding model is the first load-bearing decision in a RAG pipeline, and the default (OpenAI text-embedding-3-small) is often wrong for reasons teams discover late. Here are the four contenders I evaluate on every new build.

Model	Best for	Multilingual	Max tokens	Self-hostable	Opinion
OpenAI text-embedding-3-small	English-heavy, fast start	Weak	8,191	No	Default when corpus is English and latency matters. Cheap, good enough, zero setup.
Cohere Embed v3	Multilingual production	Strong (100+ languages)	512	No	Pick this when corpus has meaningful non-English content and you’re staying on managed infra.
BGE-M3 (open source)	Self-hosted, multilingual	Strong	8,192	Yes	Our pick for air-gapped or cost-sensitive builds. MTEB-competitive and runs on modest GPUs.
Voyage AI voyage-3	Domain-specific (legal, finance, code)	Mixed	32,000	No	Worth testing when your corpus has heavy domain vocab and you’ve already ruled out OpenAI on recall.

Don’t pick blind. The MTEB leaderboard ranks embedding models across retrieval, classification, and clustering benchmarks, and you should at minimum check how your shortlist performs on the task category closest to yours (retrieval for most RAG).

Where the English-first default loses recall

Mixed-language corpora are where the OpenAI default does the most damage. English-trained embeddings produce cross-lingual semantic drift: the query arrives in one language, the candidate chunk sits in another, and both get scored in a vector space that clusters English vocabulary far more tightly than anything else. A French-language query about a 2019 regulatory circular can pull back an English commentary on an unrelated rule, and nothing in your logs will tell you why. Recall looks healthy on the English test set and falls apart in the language your users actually type in.

The rule I use: if more than roughly 20% of your corpus is non-English, don’t default to an English-first embedding model. Test at least one multilingual option on a representative query set, in the language the questions will arrive in, before you lock the choice in.

The version drift failure mode

One detail that bites teams six months in: embedding models version. OpenAI, Cohere, and most managed providers ship updates. If you re-embed a subset of your corpus with a newer model version, the vector space shifts. Queries embedded against v2 get scored against chunks embedded in v1 and v2 mixed together, and retrieval quality degrades in ways that look like random noise.

Two fixes. Version-lock the model (pin to a specific version string, never auto-upgrade), or re-embed the entire corpus whenever you change models. We cover this more in the failure mode taxonomy below.

Decision 2: Chunking strategy

The decision isn’t chunk size, it’s chunk boundary

Most chunking guides obsess over size. 256 tokens? 512? 1,024? The more important decision is where the boundary falls. A 512-token chunk that splits a safety procedure across its midpoint is worse than an 800-token chunk that keeps the procedure intact.

Strategy	When to use	Trade-off
Fixed-size	Fast prototype, uniform docs	Breaks structured content. Our least-favorite default.
Structure-aware	Documents with clear hierarchy (manuals, contracts, legal)	Requires a parser. Worth the effort.
Parent-child	Long docs where retrieval and context need different granularities	More complex indexing, better precision+context balance.
Semantic chunking	Unstructured prose	Depends on segmentation model quality. Test before committing.

Why fixed-size chunking passes the benchmark and fails the user

Fixed-size chunking survives benchmark queries and dies on the ones that matter. Ask a corpus of technical manuals “what’s the lockout procedure for a hydraulic tensioner?” and you get the right manual back with a chunk that starts mid-procedure, drops the prerequisites, or stops two steps before the end.

Retrieval did its job. Chunking handed over partial context. Someone reading half a safety procedure is worse off than someone who got nothing at all, because they might act on it.

The fix is a parser that understands document structure. Azure Document Intelligence, Unstructured, and LlamaParse all read section headers, procedure numbering, and clause boundaries, which lets you treat a procedure as an atomic unit even when that pushes a chunk well past your target size. Chunk size is a preference. An intact procedure is a requirement.

LlamaIndex chunk size study

LlamaIndex published a chunk size study where 512-token chunks came out ahead on their test corpus. The finding gets cited as if it’s universal. Their corpus isn’t yours. The value of that study is the methodology (test multiple chunk sizes against a real query set and measure both retrieval and generation quality), not the specific number.

Run the same test on your corpus. You might land on 384 or 800 depending on document density.

Decision 3: Retrieval pattern

The decision matrix nobody publishes

There are four retrieval patterns worth considering. Most guides describe them. Few give you the decision matrix.

Pattern	When it wins	Latency cost	Failure signature	Self-hostable
BM25 (keyword)	Exact-match queries: part numbers, citations, codes, acronyms	Baseline (lowest)	Fails on synonyms, paraphrasing, conceptual queries	Yes (Elasticsearch, OpenSearch, Tantivy)
Dense (vector)	Conceptual and semantic queries, paraphrased natural language	Baseline + vector search	Fails on exact-match, out-of-vocabulary terms, rare entities	Yes (Qdrant, Weaviate, Milvus)
Hybrid (RRF fusion)	Production default. Most corpora have both exact and conceptual queries	+30-50ms over single retriever	Inherits weaknesses when both retrievers fail on same query	Yes
Reranking (cross-encoder)	Accuracy matters more than latency, top-k needs precision boost	+100-300ms	Can amplify retrieval errors if top-k is already bad	Partial (Cohere Rerank is managed; BGE reranker self-hostable)

BM25 wins when a user asks for “part number 4521-A” or “section 3.2(b)” or “code C-7.” Dense embeddings fragment those tokens and miss the match. BM25 also wins on any query with a specific rare term that the embedding model hasn’t seen enough of during training.

Dense wins when the query is “how do I handle a pressure drop during descent” and the document says “when pressure decreases on the way down, operators should.” Exact-match retrieval misses that; dense embeddings get it.

Hybrid with Reciprocal Rank Fusion is the default production choice. It combines both retrievers and works well when your query distribution has a mix of exact and conceptual questions, which is almost always true. For a deeper framework comparison of LangChain vs LlamaIndex and their retrieval orchestration trade-offs, see our dedicated post.

Reranking is where the latency conversation gets interesting.

When reranking earns its latency

Add a cross-encoder when the highest-similarity chunk is routinely not the most relevant one. Regulated corpora are the clearest case. A circular from 2020 can score high on vector similarity against a 2023 query even though a 2022 amendment supersedes it, and cosine distance has no way of knowing one document retired the other. A reranker scores relevance rather than raw similarity, so the superseding document comes back on top.

The usual shape is rerank the top 10, pass the top 3 to generation. That extra 150-200ms is easy to defend when a wrong answer produces a compliance finding or an invoice dispute. It’s much harder to defend on an internal search box where the user can simply ask again.

When to skip the reranker

Real-time systems are the obvious exception. If retrieval fires inside a loop that has to answer in a few hundred milliseconds, 150-300ms of reranking is the entire budget. Air-gapped deployments hit a second wall: the managed rerankers are API calls, and the self-hostable ones want GPU you may not be allowed to ship to the site.

Tune the hybrid retriever instead. Push BM25 weights harder toward exact match when the domain is heavily codified: rule numbers, part codes, regulation citations, anything where the user types an identifier rather than a description. You give up some conceptual recall and get the latency budget back.

Retrieval pattern is a quality-versus-latency trade.

Decision 4: LLM selection and context assembly

The “Lost in the Middle” problem

Before you pick an LLM, read Liu et al., TACL 2023. The paper documents a U-shaped accuracy curve: LLMs perform significantly worse on information placed in the middle of long contexts than on information at the beginning or end. The effect holds even in long-context models that advertise 128K or 1M windows.

The architectural implication is simple and widely ignored. Ranked order is the wrong order. Sort your top-k by descending score and the strongest chunk lands at position 1, the weakest lands at position k, and everything in between falls into the middle of the window. Once k gets past three or four, that middle band is holding most of your usable evidence, and it’s the band the model reads least reliably.

Restructure context assembly so the highest-relevance chunks sit at the start and end of the context window, with medium-relevance in the middle. Also reduce k. A top-5 assembly with thoughtful ordering outperforms a top-15 dump every time we’ve tested it.

LLM selection table

Model	Context window	Best for	Offline	Opinion
GPT-5.6 Sol	1.05M	Complex reasoning, general-purpose default	No	Where we start when no other constraint binds.
GPT-5.6 Luna	1.05M	High-volume extraction and summarisation	No	The cheap route for the majority of RAG calls, which are not hard.
Claude Sonnet 5	1M	Synthesis across dense documents	No	Pick it when the answer has to integrate evidence from many chunks rather than quote one.
Gemini 3.7 Flash	1M	Long context at low cost per token	No	Worth benchmarking against Luna on your own corpus. The gap between them moves.
Qwen 3.x / Llama 4 (open source)	128K to 10M	Offline, air-gapped, data-residency constrained	Yes	Our pick when the system cannot call a managed API. Quality has closed most of the gap for retrieval-grounded answering.

Model lineup checked against vendor documentation in August 2026. Context windows and tiers move every few months, so verify against current docs before you commit a routing design to one of these names.

Multi-LLM routing and when it’s worth the complexity

Routing across several models works when every route earns its slot. Classify the query at intake, then send plain fact retrieval to the cheap fast model, long-context synthesis to the model that integrates well across many documents, oversized contexts to the widest window you can get, and extended chain-of-thought work to a reasoning model. If you can’t name the task category a route serves and measure it, you’ve bought four sets of API keys, four rate limits, and four failure modes to run one system.

Start with one model. Add a second only when a measurable task category consistently underperforms and your eval numbers show it.

Context assembly rules in practice

Three rules I apply on every production build:

Order the context with highest-confidence chunks first and last, medium in the middle. This counters Lost in the Middle without adding latency.

Embed source citations inside each chunk during indexing, not only at generation time. If the chunk text itself carries “Source: Manual 4521-A, Section 3.2, Rev. 2023-08,” the LLM can cite reliably in the output and you can audit faithfulness against ground truth.

Cap the context at around 70% of the model’s window. You need headroom for system prompts, tool instructions, chain-of-thought tokens, and the output itself. Packing to 95% of the window is how you get truncation bugs in production.

Decision 5: Evaluation pipeline

What you need to measure (and tooling)

Tool	Measures	Integration	Opinion
RAGAS	Retrieval recall, faithfulness, answer relevance	Python, LangChain-native	Our default. Wide coverage, minimal setup.
TruLens	Feedback functions, agent traces	Python, framework-agnostic	Pick for agentic RAG where you need trace-level visibility.
LlamaIndex Eval	Retrieval + response quality	LlamaIndex-native	Good if you’re already on LlamaIndex.
DeepEval	CI/CD-oriented testing, pytest-style	Python, CI-friendly	Use this to gate deploys. Runs in your pipeline.
Custom QA suite	Ground-truth correctness	Manual	Always build one. 100-200 human-written pairs beat any framework metric.

Start with RAGAS for most projects. Add TruLens when you move to agentic architectures. Bolt DeepEval into CI/CD when evaluation becomes a deploy gate.

Three metrics that matter before the rest

Track these before anything else.

Retrieval recall: did the right chunks land in the top-k? You cannot diagnose generation quality until you know whether retrieval is delivering the correct context. Measure this first, in isolation.

Faithfulness: did the LLM answer from the retrieved context, or did it hallucinate around it? A faithfulness score below ~0.9 means your system is inventing, and no amount of retrieval tuning fixes that.

Ground-truth answer correctness: build 100-200 human-written question-answer pairs covering your highest-volume query patterns, and run them on every deploy. This is the slowest metric to build and the one that catches the most real failures.

Wire the suite into CI before you need it

An eval suite nobody blocks a deploy on is a dashboard. Set explicit thresholds on retrieval recall and faithfulness, run the suite on every PR, and fail the build when either metric drops below its line.

Set those thresholds per query category as well as on the average. The regressions that hurt most are the ones that look like wins: a chunk-size tune that lifts the mean across the whole test set while gutting recall on the small, high-criticality slice where a wrong answer costs something real. A per-category floor catches that on the PR that caused it.

Failure mode taxonomy

Six failure modes cover almost every broken RAG system I’ve audited. Each is diagnosable if you log per-stage outputs (retrieval candidates, reranked order, context sent to LLM, final generation). If those four layers aren’t logged separately, you can’t tell which one is failing.

Failure Mode	Observable Symptom	Root Cause	Fix
Chunking boundary split	Correct document cited, partial or truncated answer	Fixed-size chunk split a procedure, clause, or logical unit	Structure-aware chunking with a document parser
Embedding domain mismatch	Low recall on industry terms, acronyms, or non-English queries	Embedding model not trained on domain vocab or language	Domain-specific or multilingual embedding model
Context utilization (Lost in the Middle)	Right chunks retrieved, answer misses middle-position info	LLM attention degrades for mid-position context	Restructure: high-relevance at ends, reduce k
Index staleness	Confidently wrong answer citing an outdated document	Source doc updated but index not refreshed	Incremental indexing plus freshness checks per source
Embedding version drift	Retrieval quality degrades after partial re-indexing	Mixed embedding model versions in same index	Version-lock the model or re-embed the full corpus
Query-time latency blowout	p95 latency spikes 2-5x over p50	Chained synchronous retrieval, rerank, and LLM calls without caching	Semantic caching, parallelize independent stages, enforce latency budgets

None of these show up on a demo query. Stress-test with a realistic query distribution before launch.

If you’re auditing a RAG stack or designing a new one, we can do an architecture review before you commit to a vendor or an approach. The goal is catching the decisions that are expensive to reverse, while they’re still cheap to change. Get in touch. If you’re still pricing the build, how we scope and price the work is the place to start. If you’re deciding between building and buying, start with the build vs buy decision framework.

Frequently Asked Questions

BM25 vs dense vs hybrid: which for production?

Hybrid with Reciprocal Rank Fusion, as a default. Nearly every production corpus has a mix of exact-match queries (part numbers, citations, codes) and conceptual queries (paraphrased natural language), and hybrid handles both with a 30-50ms latency cost over a single retriever. Go pure BM25 only if your queries are almost entirely exact-match. Go pure dense only if you’ve tested and confirmed BM25 adds noise on your query distribution.

When should I add a reranker?

When precision matters more than 100-300ms of added latency, and when your top-k retrieval has enough candidates that reordering meaningfully improves top-3 quality. Regulated corpora, where a stale document outranks the one that superseded it, usually justify the cost. Real-time and offline systems usually can’t afford it. Test with a ground-truth query set before committing: if rerank doesn’t move your top-3 precision by more than 5-10 points, the latency isn’t worth it.

How does “Lost in the Middle” affect production RAG?

The Liu et al. paper shows LLMs perform worse on information in the middle of long contexts, with a U-shaped accuracy curve that holds even in long-context models. In practice: don’t pass top-k chunks to the LLM in ranked order. Restructure so the highest-relevance chunks are at the start and end, medium in the middle. And reduce k. A tight top-5 with thoughtful ordering beats a top-15 dump.

Which embedding model for multilingual corpora?

If more than ~20% of your corpus is non-English, don’t default to an English-first model like OpenAI text-embedding-3-small. Options include Cohere Embed v3 (managed, strong multilingual) or BGE-M3 (open source, self-hostable, MTEB-competitive). Test both against a representative query set in the non-English language before locking the choice in, and measure cross-lingual recall specifically. That’s the number an English-first model costs you, and it’s the one that never shows up on an English test set.

How do I know if my eval suite is catching real failures?

Three tests. One: is your eval suite blocking deploys, or is it a dashboard nobody reads? If it’s the latter, it’s not doing its job. Two: does it measure retrieval recall, faithfulness, and ground-truth answer correctness separately, so you can localize failures to the right stage? Three: does it cover your highest-volume production query patterns, or only the queries you tested during development? If you can’t answer yes to all three, your suite is missing real failures.