RAG Architecture Explained: Components, Patterns, and Production Considerations

Five architectural decisions determine whether a RAG system works in production. I’ve watched teams make all five correctly and ship something that helps engineers find safety procedures in under 90 seconds. I’ve watched teams make two of them wrong and spend six months debugging a pipeline that was architecturally broken from week one. The gap isn’t talent. It’s that the wrong decisions at the retrieval, embedding, or context-assembly layer compound invisibly until production traffic exposes them.

This post assumes you already know. It’s for the engineer, tech lead, or AI architect who needs to decide how to build it: which embedding model, which chunking boundary, which retrieval pattern, which LLM, and how to evaluate the whole thing before it goes live. Five decisions, each anchored in systems we’ve shipped: the offshore energy vessel management build, the Italian legal research build, and the offline maritime simulation build.

Decision 1: Which embedding model

What you’re actually choosing between

The embedding model is the first load-bearing decision in a RAG pipeline, and the default (OpenAI text-embedding-3-small) is often wrong for reasons teams discover late. Here are the four contenders I evaluate on every new build.

Model	Best for	Multilingual	Max tokens	Self-hostable	Opinion
OpenAI text-embedding-3-small	English-heavy, fast start	Weak	8,191	No	Default when corpus is English and latency matters. Cheap, good enough, zero setup.
Cohere Embed v3	Multilingual production	Strong (100+ languages)	512	No	Pick this when corpus has meaningful non-English content and you’re staying on managed infra.
BGE-M3 (open source)	Self-hosted, multilingual	Strong	8,192	Yes	Our pick for air-gapped or cost-sensitive builds. MTEB-competitive and runs on modest GPUs.
Voyage AI voyage-3	Domain-specific (legal, finance, code)	Mixed	32,000	No	Worth testing when your corpus has heavy domain vocab and you’ve already ruled out OpenAI on recall.

Don’t pick blind. The MTEB leaderboard ranks embedding models across retrieval, classification, and clustering benchmarks, and you should at minimum check how your shortlist performs on the task category closest to yours (retrieval for most RAG).

The legal research build: why we didn’t default to OpenAI embeddings

On a legal research build for an Italian tax and legal advisory firm, we were working with a 25GB corpus of decades of legal documents, case law, regulatory texts, and circulars. Most of it was Italian. Some was English. The naive move is to pick OpenAI embeddings and move on.

We didn’t. English-trained embeddings produce cross-lingual semantic drift when the query is in one language and the candidate chunk is in another. An Italian query about a 2019 circular might pull back an English-language commentary on an unrelated regulation because the English embedding space has tighter clustering for English vocabulary. The multilingual embedding choice was load-bearing here. Cross-lingual recall matters when your lawyers are asking questions in Italian against documents that exist in both languages.

The rule I use now: if more than roughly 20% of your corpus is non-English, don’t default to an English-first embedding model. Test at least one multilingual option on a representative query set before you lock the choice in.

The version drift failure mode

One detail that bites teams six months in: embedding models version. OpenAI, Cohere, and most managed providers ship updates. If you re-embed a subset of your corpus with a newer model version, the vector space shifts. Queries embedded against v2 get scored against chunks embedded in v1 and v2 mixed together, and retrieval quality degrades in ways that look like random noise.

Two fixes. Version-lock the model (pin to a specific version string, never auto-upgrade), or re-embed the entire corpus whenever you change models. Halfway doesn’t work. We cover this more in the failure mode taxonomy below.

Decision 2: Chunking strategy

The decision isn’t chunk size, it’s chunk boundary

Most chunking guides obsess over size. 256 tokens? 512? 1,024? The more important decision is where the boundary falls. A 512-token chunk that splits a safety procedure across its midpoint is worse than an 800-token chunk that keeps the procedure intact.

Four strategies worth knowing:

Strategy	When to use	Trade-off
Fixed-size	Fast prototype, uniform docs	Breaks structured content. Our least-favorite default.
Structure-aware	Documents with clear hierarchy (manuals, contracts, legal)	Requires a parser. Worth the effort.
Parent-child	Long docs where retrieval and context need different granularities	More complex indexing, better precision+context balance.
Semantic chunking	Unstructured prose	Depends on segmentation model quality. Test before committing.

The offshore energy build: 15,000 manuals and the boundary problem

On the vessel management build for an offshore energy client, the team needed to pull answers from 15,000+ technical manuals spread across multiple SharePoint sites controlled by different regional IT teams. Engineers were taking 23+ minutes to find a procedure. The target was under 90 seconds.

Our first chunking pass was fixed-size. It worked on benchmark queries but failed on the ones that mattered. A question like “what’s the lockout procedure for a hydraulic tensioner?” would retrieve the right manual but return a chunk that started mid-procedure, missed the prerequisites, or cut off before the final steps. The retrieval layer found the right document. The chunking layer delivered partial context. An engineer reading a half-procedure on a rig is worse off than one who gets nothing, because they might act on it.

We switched to Azure Document Intelligence for document-structure-aware chunking. The parser reads section headers, procedure numbering, and clause boundaries, and it treats a safety procedure as an atomic unit even if that means a chunk runs longer than our default. Results were materially better on procedure-level queries. We stopped splitting atomic safety units across chunk boundaries, and that single change moved the system from “finds the doc” to “delivers the answer.”

LlamaIndex chunk size study

LlamaIndex published a chunk size study where 512-token chunks came out ahead on their test corpus. The finding gets cited as if it’s universal. It isn’t. Their corpus isn’t yours. The value of that study is the methodology (test multiple chunk sizes against a real query set and measure both retrieval and generation quality), not the specific number.

Run the same test on your corpus. You might land on 384 or 800 depending on document density.

Decision 3: Retrieval pattern

The decision matrix nobody publishes

There are four retrieval patterns worth considering. Most guides describe them. Few give you the decision matrix.

Pattern	When it wins	Latency cost	Failure signature	Self-hostable
BM25 (keyword)	Exact-match queries: part numbers, citations, codes, acronyms	Baseline (lowest)	Fails on synonyms, paraphrasing, conceptual queries	Yes (Elasticsearch, OpenSearch, Tantivy)
Dense (vector)	Conceptual and semantic queries, paraphrased natural language	Baseline + vector search	Fails on exact-match, out-of-vocabulary terms, rare entities	Yes (Qdrant, Weaviate, Milvus)
Hybrid (RRF fusion)	Production default. Most corpora have both exact and conceptual queries	+30-50ms over single retriever	Inherits weaknesses when both retrievers fail on same query	Yes
Reranking (cross-encoder)	Accuracy matters more than latency, top-k needs precision boost	+100-300ms	Can amplify retrieval errors if top-k is already bad	Partial (Cohere Rerank is managed; BGE reranker self-hostable)

Quick guidance:

BM25 wins when a user asks for “part number 4521-A” or “section 3.2(b)” or “code C-7.” Dense embeddings fragment those tokens and miss the match. BM25 also wins on any query with a specific rare term that the embedding model hasn’t seen enough of during training.

Dense wins when the query is “how do I handle a pressure drop during descent” and the document says “when pressure decreases on the way down, operators should.” Exact-match retrieval misses that; dense embeddings get it.

Hybrid with Reciprocal Rank Fusion is the default production choice. It combines both retrievers and works well when your query distribution has a mix of exact and conceptual questions, which is almost always true. For a deeper framework comparison of LangChain vs LlamaIndex and their retrieval orchestration trade-offs, see our dedicated post.

Reranking is where the latency conversation gets interesting.

Reranking decision on the legal corpus

The legal research build runs an AgenticRAG system with anomaly detection across Italian legal corpora. We use Cohere Rerank on top-10 retrieved chunks before passing top-3 to the generation model. The reason is specific to the domain: in legal retrieval, the highest-confidence retrieval chunk is often not the most relevant. A circular from 2020 might score high on vector similarity for a 2023 query even though a 2022 amendment supersedes it. The reranker, trained on relevance rather than just similarity, consistently surfaces the right chunk.

Reranking earned its latency cost here. We saw clear precision gains on ground-truth legal queries, and the extra 150-200ms was acceptable given how much a wrong legal answer costs downstream.

The offline maritime build: latency-constrained retrieval

The maritime simulation build is a training platform running fully offline with 0 cloud dependencies. 250+ rules across 6 categories, 4 specialized agents. Qwen runs locally. Qdrant runs locally. Every millisecond counts because the system feeds real-time rule monitoring during simulation runs.

We didn’t add a reranker. 150-300ms of added latency was unacceptable for a system that needs to fire rule checks mid-simulation. Instead we tuned BM25 weights in the hybrid retriever more aggressively toward exact-match because maritime rules are heavily codified (rule numbers, vessel codes, regulation citations). Pure hybrid, no rerank, tuned for the query distribution.

The broader lesson: retrieval pattern is a quality-versus-latency trade. Know your constraint before you pick. An offline real-time system and a legal research agent sit at opposite ends of that trade-off and deserve different architectures.

Decision 4: LLM selection and context assembly

The “Lost in the Middle” problem

Before you pick an LLM, read Liu et al., TACL 2023. The paper documents a U-shaped accuracy curve: LLMs perform significantly worse on information placed in the middle of long contexts than on information at the beginning or end. The effect holds even in long-context models that advertise 128K or 1M windows.

The architectural implication is simple and widely ignored. Don’t pass your top-k chunks to the LLM in ranked order, because that puts your highest-confidence evidence at position 1 and your second-best at position 2, and so on, with the weakest evidence in the middle, which is exactly where attention degrades the least on your best chunks and the most on your marginal ones.

The fix: restructure context assembly so the highest-relevance chunks sit at the start and end of the context window, with medium-relevance in the middle. Also reduce k. A top-5 assembly with thoughtful ordering outperforms a top-15 dump every time we’ve tested it.

LLM selection table

Model	Context window	Best for	Offline	Opinion
GPT-4o	128K	General-purpose, cost-efficient at scale	No	Our default when none of the other constraints bind.
Claude 3.5 Sonnet	200K	Long-context synthesis, reasoning over dense docs	No	Pick when the retrieved context is long and the answer requires integration.
Gemini 1.5 Pro	1M-2M	Edge cases with very long context	No	Reach for this when reducing k isn’t feasible. Rare but real.
Qwen 2.5 / Llama 3.3 (open source)	32K-128K	Offline, air-gapped, cost-sensitive	Yes	Our pick when the system can’t call managed APIs. Quality has closed most of the gap for RAG.

Multi-LLM routing (the legal research build)

The legal research build routes across four models: GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and DeepSeek R1. This isn’t complexity theater. Each model earned its slot for a specific task type. Query classification at intake routes the request: straightforward fact retrieval goes to GPT-4o, long-context synthesis over multiple legal documents goes to Claude, edge cases with very large context go to Gemini, and deep reasoning tasks where we need extended chain-of-thought go to DeepSeek R1.

The design principle: start with one model, add another only when a measurable task category consistently underperforms. Multi-LLM architectures should grow from evidence, not from the assumption that more models equals better results.

Context assembly rules in practice

Three rules I apply on every production build:

Order the context with highest-confidence chunks first and last, medium in the middle. This counters Lost in the Middle without adding latency.

Embed source citations inside each chunk during indexing, not only at generation time. If the chunk text itself carries “Source: Manual 4521-A, Section 3.2, Rev. 2023-08,” the LLM can cite reliably in the output and you can audit faithfulness against ground truth.

Cap the context at around 70% of the model’s window. You need headroom for system prompts, tool instructions, chain-of-thought tokens, and the output itself. Packing to 95% of the window is how you get truncation bugs in production.

Decision 5: Evaluation pipeline

What you need to measure (and tooling)

Tool	Measures	Integration	Opinion
RAGAS	Retrieval recall, faithfulness, answer relevance	Python, LangChain-native	Our default. Wide coverage, minimal setup.
TruLens	Feedback functions, agent traces	Python, framework-agnostic	Pick for agentic RAG where you need trace-level visibility.
LlamaIndex Eval	Retrieval + response quality	LlamaIndex-native	Good if you’re already on LlamaIndex.
DeepEval	CI/CD-oriented testing, pytest-style	Python, CI-friendly	Use this to gate deploys. Runs in your pipeline.
Custom QA suite	Ground-truth correctness	Manual	Always build one. 100-200 human-written pairs beat any framework metric.

Start with RAGAS for most projects. Add TruLens when you move to agentic architectures. Bolt DeepEval into CI/CD when evaluation becomes a deploy gate.

Three metrics that matter before the rest

If you can only track three things, track these.

Retrieval recall: did the right chunks land in the top-k? You cannot diagnose generation quality until you know whether retrieval is delivering the correct context. Measure this first, in isolation.

Faithfulness: did the LLM answer from the retrieved context, or did it hallucinate around it? A faithfulness score below ~0.9 means your system is inventing, and no amount of retrieval tuning fixes that.

Ground-truth answer correctness: build 100-200 human-written question-answer pairs covering your highest-volume query patterns, and run them on every deploy. This is the slowest metric to build and the one that catches the most real failures.

The offshore energy build: evaluation as deployment gate

On the vessel management build, we built a test suite covering the highest-volume query patterns from the engineering teams who actually use the system. Retrieval recall and faithfulness thresholds act as deployment gates. A PR that regresses either metric below its threshold fails the pipeline and doesn’t merge. This gate caught regressions before prod on multiple occasions, including one where a well-intentioned chunk-size tune improved average performance but degraded recall on the highest-criticality safety queries.

If your evaluation suite isn’t blocking deploys, it’s decoration.

Failure mode taxonomy

Six failure modes cover almost every broken RAG system I’ve audited. Each is diagnosable if you log per-stage outputs (retrieval candidates, reranked order, context sent to LLM, final generation). If those four layers aren’t logged separately, you can’t tell which one is failing.

Failure Mode	Observable Symptom	Root Cause	Fix
Chunking boundary split	Correct document cited, partial or truncated answer	Fixed-size chunk split a procedure, clause, or logical unit	Structure-aware chunking with a document parser
Embedding domain mismatch	Low recall on industry terms, acronyms, or non-English queries	Embedding model not trained on domain vocab or language	Domain-specific or multilingual embedding model
Context utilization (Lost in the Middle)	Right chunks retrieved, answer misses middle-position info	LLM attention degrades for mid-position context	Restructure: high-relevance at ends, reduce k
Index staleness	Confidently wrong answer citing an outdated document	Source doc updated but index not refreshed	Incremental indexing plus freshness checks per source
Embedding version drift	Retrieval quality degrades after partial re-indexing	Mixed embedding model versions in same index	Version-lock the model or re-embed the full corpus
Query-time latency blowout	p95 latency spikes 2-5x over p50	Chained synchronous retrieval, rerank, and LLM calls without caching	Semantic caching, parallelize independent stages, enforce latency budgets

Each is preventable with the right logging and the right architectural choice upstream. None of them show up on a demo query. All of them show up under production traffic, which is why I push teams to stress-test with realistic query distribution before launch, not after.

If you’re auditing a RAG stack or designing a new one, we can do an architecture review before you commit to a vendor or an approach. The goal is catching the load-bearing decisions early, while they’re still cheap to change. Get in touch. If you’re still pricing the build, how we scope and price the work is the place to start. If you’re deciding between building and buying, the build vs buy decision framework is the place to start.

Frequently Asked Questions

BM25 vs dense vs hybrid: which for production?

Hybrid with Reciprocal Rank Fusion, as a default. Nearly every production corpus has a mix of exact-match queries (part numbers, citations, codes) and conceptual queries (paraphrased natural language), and hybrid handles both with a 30-50ms latency cost over a single retriever. Go pure BM25 only if your queries are almost entirely exact-match. Go pure dense only if you’ve tested and confirmed BM25 adds noise on your query distribution.

When should I add a reranker?

When precision matters more than 100-300ms of added latency, and when your top-k retrieval has enough candidates that reordering meaningfully improves top-3 quality. We added reranking on the legal research build because legal precision justified the cost. We skipped it on the maritime simulation build because the system is real-time and offline. Test with a ground-truth query set before committing: if rerank doesn’t move your top-3 precision by more than 5-10 points, the latency isn’t worth it.

How does “Lost in the Middle” affect production RAG?

The Liu et al. paper shows LLMs perform worse on information in the middle of long contexts, with a U-shaped accuracy curve that holds even in long-context models. In practice: don’t pass top-k chunks to the LLM in ranked order. Restructure so the highest-relevance chunks are at the start and end, medium in the middle. And reduce k. A tight top-5 with thoughtful ordering beats a top-15 dump.

Which embedding model for multilingual corpora?

If more than ~20% of your corpus is non-English, don’t default to an English-first model like OpenAI text-embedding-3-small. Options include Cohere Embed v3 (managed, strong multilingual) or BGE-M3 (open source, self-hostable, MTEB-competitive). Test both against a representative query set in the non-English language before locking the choice in. On the legal research build, the multilingual embedding was load-bearing for cross-lingual recall.

How do I know if my eval suite is catching real failures?

Three tests. One: is your eval suite blocking deploys, or is it a dashboard nobody reads? If it’s the latter, it’s not doing its job. Two: does it measure retrieval recall, faithfulness, and ground-truth answer correctness separately, so you can localize failures to the right stage? Three: does it cover your highest-volume production query patterns, or only the queries you tested during development? If you can’t answer yes to all three, your suite is missing real failures.