RAG in 2026: Contextual Retrieval, Hybrid Search and Agentic RAG

A RAG demo works on the first afternoon. You load a folder of PDFs, ask a question, and get a fluent answer with a quote. The trouble starts a month later, when someone asks about part number TS-999, or about "last year's policy", or about a contract the asker should not see. This guide is about what happens between the demo and a system people trust. It is written for two readers: leaders who need to judge a RAG project, and engineers who need to build one.

You will get a named model for the work (we call it the Retrieval Ladder), two programs we ran offline with their real output, a table of common failures with a test for each, a section on document permissions, a cost worked out from published figures, and a production checklist. Where a number comes from a vendor or a paper, we name the source and the date we read it, and say what we make of it. Where a statement is our own judgement, we label it as ours.

This guide covers the techniques. For the system around them, such as ingestion, access layers and what a build costs, read our companion post on enterprise RAG architecture. The two are meant to be read together, and this one does not repeat that one.

What RAG is and where it breaks

Retrieval-augmented generation, or RAG, joins two kinds of memory. The model holds what it learned in training. A search index holds your documents. The original 2020 paper described this as combining a pre-trained generator with a dense vector index that a neural retriever reads. Six years on, the pattern is the same. Only the parts have improved.

The basic loop

A RAG system does five things for every question.

1. It turns the question into a search.
2. It finds passages that might answer it.
3. It puts the best passages into the prompt.
4. It asks the model to answer from those passages and cite them.
5. It returns the answer with the sources.

Before any of that, an offline job splits your documents into chunks, turns each chunk into a vector (a list of numbers that stands for its meaning), and stores both the text and the vector in an index. We call that the ingestion side. Everything above is the query side.

Why retrieval is the weak link

If the right passage is in the prompt, a good model usually answers well. If it is not, no model can fix that. It will either say it does not know, or it will write something that sounds right. So most RAG quality work is retrieval work.

This is our reading of the evidence, and the evidence is indirect. What we checked: Stanford's study of commercial legal research tools that use RAG, read on 2026-10-27. The researchers found the tools still hallucinated more than 17% of the time on their benchmark, with one tool above 34%. They reported that the tools beat general chatbots, and that RAG is not a cure. They pointed to hard retrieval, retrieved authority that no longer applies, and tools that accept a false premise from the user. What we see in it: two of the three causes, hard retrieval and inapplicable sources, are retrieval problems. That matches how we triage a bad answer.

Our recommendation for a business reader is practical. When a RAG system gives a wrong answer, ask first what it retrieved. If a vendor cannot show you the retrieved passages, we would not accept the system for production.

The Retrieval Ladder

Teams often fix RAG by changing the model, the prompt, or the vector database. Those are the expensive moves, and they often miss the cause. We recommend a different habit, and it is how we plan RAG work: climb a short ladder one rung at a time, and measure after each rung. The ladder is our own framework, not a standard, and the order reflects cost and risk as we judge them.

The rest of this guide goes rung by rung. Rung 1 gets its own section near the end, because it needs the most care, but you do it first.

Chunking and metadata

A chunk is the unit your system retrieves. It is also the unit the model reads. Get it wrong and everything after it suffers.

Chunk size and boundaries

In Anthropic's post, which we read on 2026-10-27, chunk size, chunk boundary and chunk overlap are all named as things that can change retrieval performance. We found no published rule that gives the right size for every corpus, so treat any number you read, including ours, as a starting point to test.

Our engineering judgment is that the boundary matters more than the size. Cut a clause in half and you get a chunk that reads like a rule but lacks its exception. The table lists the common choices.

The last row is where we expect many projects to stall. In the Manual Minds AI platform, which is a published EdgeFirm case study, the manuals carry key facts in tables and diagrams. As the case study reports it, the pipeline pulls text, tables and diagram images separately, describes every image with a vision model, and stores each piece with its file and page reference. That work is dull, and in our judgment it decides the result.

Metadata that earns its place

Metadata is the label on each chunk. Four labels pay for themselves in almost every corpus.

• Source and page. You need these for citations.
• Date and version. Without them the search cannot prefer this year's policy over the old one.
• Document type. A contract and a chat message deserve different handling.
• Access rule. Who may read this. We return to it in the permissions section.

We would add a fifth when it applies: the tenant or customer the chunk belongs to. The Manual Minds case study states that each machine gets its own isolated collection, so a question about one machine is answered only from that machine's manual.

Embeddings and dense search

An embedding model turns a piece of text into a vector. Texts with similar meaning land close together. Dense search compares the question vector to chunk vectors and returns the nearest ones.

What dense search is good at

Dense search matches meaning. A person asks "how do I get my money back" and the policy says "refund to the original payment method". The two share no words, but their vectors are close. Keyword search would miss this. Our offline sample below reproduces that case, and in our run keyword search returned nothing for it.

Where dense search is weak

Vectors blur exact strings. Anthropic's post gives the example of an error code: a query for TS-999 may bring back content about error codes in general and miss the exact match. Our read is that part numbers, clause numbers, names and acronyms behave the same way. This is why we do not recommend dense search alone.

Choosing an embedding model

In its original contextual retrieval tests, Anthropic tried several embedding models and reported that Gemini and Voyage embeddings were the most effective. That is one vendor's result on its own datasets. We do not rank embedding models here, because the ranking depends on your language, your domain and your document length, and it changes often. Pick two or three candidates, run your test set against each, and choose on recall, price and where the data may go.

Two rules we apply, both our own judgment. First, use the same model to embed documents and queries, and write down its version, because changing it means re-embedding everything. Second, if your data must stay in your network, check that the model can run there. Our guide to self-hosted LLMs covers the hardware side.

BM25, dense search and a reranker

This is rung 3 and rung 4 of the ladder, and the default design for most business corpora.

BM25 in one paragraph

BM25 is a keyword ranking formula. It scores a chunk higher when it contains the query words, more so for rare words and less so for very long chunks. It needs no model and no training. It is fast and it is exact. Anthropic's post describes it as effective for queries with unique identifiers or technical terms, which matches what we saw in our sample below.

Merging two lists with rank fusion

Keyword search and vector search return scores on different scales, so you cannot add them. Reciprocal rank fusion, or RRF, avoids the problem by using only the rank. Each list gives a chunk a score of 1 divided by (k plus its rank), and the scores are added. We read the 2009 paper that introduced it on 2026-10-27. It fixed k at 60 after a pilot study, and it reported that RRF beat the best single system and beat Condorcet Fuse, an established method for combining rankings.

RRF has no weights to tune, which is why we start with it. A chunk that ranks well in both lists rises to the top. A chunk that ranks first in only one list still survives, and that is how the exact TS-999 match is kept.

A hybrid search you can run

To see the mechanism ourselves, we wrote a small program with BM25, a toy vector space and RRF, and scored each method on five hand-made questions. The "vectors" are a toy. Words that mean the same thing share one dimension, so the question "how do I get my money back" can match a chunk about refunds. A real system uses an embedding model.

"""Offline hybrid retrieval: BM25 + toy dense vectors + reciprocal rank fusion. No network, no libraries beyond the standard library. The "dense" part is a toy:words that mean the same thing share one dimension, so a query with no word incommon with a chunk can still match it. A real system uses an embedding model."""import mathimport refrom collections import Counter CHUNKS = {    "c1": "Customers may return goods within 30 days for a refund to the original payment method.",    "c2": "Error TS-999 means the export service timed out. Restart the export job and retry.",    "c3": "Annual leave requests go to your line manager at least two weeks before the start date.",    "c4": "The Q2 2026 report says revenue grew by 3% over the previous quarter.",    "c5": "Invoices are payable within 45 days. Late payments incur a fee of 1.5% per month.",    "c6": "Staff may work remotely up to three days a week with manager approval.",} # Toy concept lexicon: each group is one dimension of the toy vector space.CONCEPTS = [    {"refund", "return", "money", "back", "reimbursement"},    {"leave", "holiday", "vacation", "time", "off"},    {"remote", "home", "wfh", "work"},    {"invoice", "payable", "payment", "pay", "bill", "late"},    {"revenue", "sales", "income", "grew", "growth"},    {"error", "timed", "timeout", "restart", "retry", "export"},] def tokens(text):    return re.findall(r"[a-z0-9][a-z0-9\-]*", text.lower()) def bm25_rank(query, docs, k1=1.5, b=0.75):    toks = {d: tokens(t) for d, t in docs.items()}    avg = sum(len(t) for t in toks.values()) / len(toks)    df = Counter(w for t in toks.values() for w in set(t))    n = len(docs)    scores = {}    for d, t in toks.items():        tf = Counter(t)        s = 0.0        for q in set(tokens(query)):            if q not in tf:                continue            idf = math.log(1 + (n - df[q] + 0.5) / (df[q] + 0.5))            s += idf * tf[q] * (k1 + 1) / (tf[q] + k1 * (1 - b + b * len(t) / avg))        scores[d] = s    return [d for d, s in sorted(scores.items(), key=lambda x: -x[1]) if s > 0] def embed(text):    words = set(tokens(text))    return [len(words & group) for group in CONCEPTS] def cosine(a, b):    na, nb = math.sqrt(sum(x * x for x in a)), math.sqrt(sum(x * x for x in b))    return sum(x * y for x, y in zip(a, b)) / (na * nb) if na and nb else 0.0 def dense_rank(query, docs):    q = embed(query)    scores = {d: cosine(q, embed(t)) for d, t in docs.items()}    return [d for d, s in sorted(scores.items(), key=lambda x: -x[1]) if s > 0] def rrf(rankings, k=60):    """Reciprocal rank fusion (Cormack et al., SIGIR 2009): sum of 1 / (k + rank)."""    scores = Counter()    for ranking in rankings:        for rank, d in enumerate(ranking, start=1):            scores[d] += 1 / (k + rank)    return [d for d, _ in scores.most_common()] # (query, the chunk a person would say answers it)TESTS = [    ("how do I get my money back", "c1"),    ("TS-999", "c2"),    ("time off request", "c3"),    ("late payment fee", "c5"),    ("work from home rules", "c6"),] def recall_at(k, ranker):    hits = sum(1 for q, gold in TESTS if gold in ranker(q)[:k])    return hits / len(TESTS) if __name__ == "__main__":    bm = lambda q: bm25_rank(q, CHUNKS)    de = lambda q: dense_rank(q, CHUNKS)    hy = lambda q: rrf([bm(q), de(q)])    print(f"{'query':<30} {'bm25':<10} {'dense':<10} {'hybrid':<10} gold")    for q, gold in TESTS:        top = lambda r: (r(q) or ["-"])[0]        print(f"{q:<30} {top(bm):<10} {top(de):<10} {top(hy):<10} {gold}")    print()    for name, r in [("bm25", bm), ("dense", de), ("hybrid", hy)]:        print(f"recall@1 {name:<7} {recall_at(1, r):.2f}   recall@3 {recall_at(3, r):.2f}")

We ran it with Python 3.13.3 on 2026-10-27. This is the real output:

query                          bm25       dense      hybrid     goldhow do I get my money back     -          c1         c1         c1TS-999                         c2         -          c2         c2time off request               -          c3         c3         c3late payment fee               c5         c5         c5         c5work from home rules           c6         c6         c6         c6recall@1 bm25    0.60   recall@3 0.60recall@1 dense   0.80   recall@3 0.80recall@1 hybrid  1.00   recall@3 1.00

A dash means the method returned nothing. What we see in the output: keyword search found no match for the money question, because no word is shared. Dense search found no match for TS-999, because the toy vector space has no dimension for it. The hybrid found both. The scores come from five questions and six chunks, so they show the mechanism and say nothing about your data.

Rerankers

A reranker is a model that reads the question and one candidate passage together and scores how well the passage answers it. It is slower than vector search, because it must read each candidate, but it is more precise. The usual pattern, and the one we recommend, is to retrieve many candidates and rerank them down to a few. In the setup Anthropic describes, it retrieved the top 150 chunks and reranked them to the top 20 with a Cohere reranker, and noted that reranking adds a small amount of latency even though the chunks are scored in parallel.

Anthropic also reported that passing 20 chunks to the model worked better than passing 10 or 5. That is a result on their data, and more chunks cost more tokens. Our read: treat 20 as a value to test, not a constant.

Contextual retrieval

Now the technique this guide is named for. It is rung 5 of the ladder, and it fixes a problem that the first four rungs cannot.

The problem: a chunk without a name

Take the chunk "The company's revenue grew by 3% over the previous quarter." Anthropic's post uses a similar example. On its own the chunk does not say which company or which quarter. A person who asks about ACME's Q2 2026 revenue will not match it by keyword, and the vector for it is generic. In our reading, whenever you split a long document, many chunks lose their name in this way. A table row loses its table title. A clause loses its contract. A step loses its procedure.

The idea

Before you index each chunk, ask a model to write a short note that places the chunk in its document. Put the note in front of the chunk. Then embed that combined text, and also add it to the BM25 index. Anthropic's version uses the full document and the chunk in the prompt and asks for "a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk." The post reports that the generated context is usually 50 to 100 tokens.

What Anthropic measured

We read Anthropic's results table on 2026-10-27. It measures the failure rate of the top 20 results, defined as one minus recall at 20, averaged across its datasets, with Gemini Text 004 embeddings.

Our read, in three points. The datasets are Anthropic's, so we would expect your numbers to differ. The article does not report contextual BM25 on its own, so we would not quote a number for it. And the figures are for one embedding model, so we treat them as a direction and not a forecast.

What it costs

The note is written once per chunk at indexing time, not at query time. Anthropic's cost estimate assumes 800-token chunks, 8,000-token documents, 50 tokens of instructions and 100 tokens of context per chunk. With prompt caching, which lets the model reuse the long document across all its chunks, it puts the one-time cost at $1.02 per million document tokens.

We checked that figure against current prices. First, our arithmetic from those assumptions, for a corpus of 5 million tokens. At 800 tokens per chunk that is 6,250 chunks. At 100 tokens of context per chunk, you add 625,000 tokens to what you embed, which is 12.5% more.

Anthropic's $1.02 figure used the prices of its small model when the post was written. We then recomputed it with the prices on the current pricing page, opened on 2026-10-27, for Claude Haiku 5.5 at prompts up to 100,000 tokens: $0.10 per million base input tokens, $0.125 for a 5-minute cache write, $0.01 for a cache read and $0.50 for output. Take one 8,000-token document cut into 10 chunks. The first call writes the document to the cache: 8,000 tokens at $0.125 per million is $0.00100. The other 9 calls read it: 72,000 tokens at $0.01 per million is $0.00072. Each call also sends about 850 uncached tokens, the chunk and the instructions: 8,500 tokens at $0.10 per million is $0.00085. Each call writes 100 tokens of context: 1,000 tokens at $0.50 per million is $0.00050. The total is $0.00307 per 8,000-token document.

That is about $0.38 per million document tokens, so the 5-million-token corpus costs about $1.92 once, against $5.10 at the old figure. Without caching, the same work would cost about $1.17 per million document tokens, or about $5.84 for the corpus, because every call would pay the base input price for the whole document. The Batch API takes 50% off input and output, which would bring the cached figure to roughly $0.19 per million. Two limits on our figure. It assumes the model writes only the note, and if you turn thinking on, thinking tokens are billed as output and the cost rises. And a mid-tier model costs more: at Claude Sonnet 5.5 prices ($2 input, $2.50 cache write, $0.10 cache read, $10 output per million) the same arithmetic gives about $6.77 per million document tokens. We would check the pricing page again before planning, because prices change.

Prompt caching matters here. Anthropic's current docs list a cache read at 0.1 times the base input price for most models (0.05 times for Claude Sonnet 5.5 and Opus 5.5), and a 5-minute cache write at 1.25 times. Cached content is kept for 5 minutes by default, and each use refreshes it. So process the chunks of one document back to back, to keep the cache warm. Short prompts are not cached at all, and the minimum length depends on the model. It is 512 tokens for Claude Haiku 5.5, so an 8,000-token document is well above it.

The prompt sample

This sample asks a small model to write the context line. The model name comes from an environment variable because model names change.

"""Contextual retrieval: ask a small model to situate each chunk in its document. NOT RUN. Needs an API key. Syntax-checked only. The prompt follows the one inAnthropic's contextual retrieval post. The model id is read from theenvironment because model names change; set CONTEXT_MODEL to a small, cheap model.The document block is marked for prompt caching, so the long document isbilled at the cache-read rate after the first chunk."""import os import anthropic client = anthropic.Anthropic()MODEL = os.environ.get("CONTEXT_MODEL", "claude-haiku-5-5") CHUNK_PROMPT = """Here is the chunk we want to situate within the whole document<chunk>{chunk}</chunk>Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else."""  def contextualize(document: str, chunk: str) -> str:    resp = client.messages.create(        model=MODEL,        max_tokens=1024,  # room for adaptive thinking on current models; the note itself is ~100 tokens        messages=[            {                "role": "user",                "content": [                    {                        "type": "text",                        "text": f"<document>\n{document}\n</document>",                        "cache_control": {"type": "ephemeral"},                    },                    {"type": "text", "text": CHUNK_PROMPT.format(chunk=chunk)},                ],            }        ],    )    # Skip any thinking block; join the text blocks.    return "".join(b.text for b in resp.content if b.type == "text").strip()  def index_text(document: str, chunk: str) -> str:    """Text to send to BOTH the embedding model and the BM25 index."""    return f"{contextualize(document, chunk)}\n{chunk}"

Note what index_text returns. The same combined text goes to the embedding model and to the BM25 index. If you only add the note to the vectors, you lose the keyword half of the gain.

The idea, run offline

For this second program we replaced the model call with a fixed lookup, then looked at what the context line does to a keyword ranking. Two filings have a chunk with identical text, and only the context says which company each came from.

"""Why a context line helps: the chunk never names the company, the context does. Offline. The "contextualizer" is a fixed lookup standing in for the model callshown in contextualize.py. Uses the BM25 function from hybrid_search.py."""from hybrid_search import bm25_rank # Three filings, each with a sentence that does not say which company it is about.CHUNKS = {    "b1": "Revenue grew by 3% over the previous quarter.",    "a1": "Revenue grew by 3% over the previous quarter.",    "b2": "Revenue fell by 2% over the previous quarter.",    "c1": "Operating costs rose by 4% over the previous quarter.",}CONTEXT = {    "b1": "From the Globex Ltd Q2 2026 filing, section on quarterly results.",    "a1": "From the ACME Corp Q2 2026 filing, section on quarterly results.",    "b2": "From the Globex Ltd Q3 2026 filing, section on quarterly results.",    "c1": "From the ACME Corp Q2 2026 filing, section on costs.",} QUERY = "ACME Corp Q2 2026 revenue growth"plain = bm25_rank(QUERY, CHUNKS)ctx = bm25_rank(QUERY, {k: f"{CONTEXT[k]} {v}" for k, v in CHUNKS.items()})print("without context:", plain)print("with context:   ", ctx)print("right answer is a1. Without context a1 and b1 have identical text, so they tie and order is arbitrary.")print("a1 is rank", plain.index("a1") + 1, "without context and rank", ctx.index("a1") + 1, "with it")

Real output from the same run:

without context: ['b1', 'a1', 'b2']with context:    ['a1', 'c1', 'b1', 'b2']right answer is a1. Without context a1 and b1 have identical text, so they tie and order is arbitrary.a1 is rank 2 without context and rank 1 with it

What we see in the output: without context, the two chunks tie, and the search has no way to prefer ACME over Globex. With context, the right one ranks first. A cost chunk from the same ACME filing also rises to second place, because it now shares the company and quarter. That side effect is the price of the method. It adds recall by adding words, and some of those words match things you did not want. This is why you still need a reranker and a test set.

GraphRAG and LazyGraphRAG

Rung 6 has two branches. This is the first.

What GraphRAG is for

Plain RAG answers questions that live in a few passages. It struggles with a question about the whole corpus, such as "What are the main themes across these 500 reports?" No single chunk holds the answer. Microsoft Research introduced GraphRAG for this kind of question, which the paper we read calls global sensemaking. An LLM first builds an entity graph from the documents, then writes summaries for groups of closely related entities, called communities. At query time each community summary produces a partial answer, and the partial answers are combined. The authors report more complete and more varied answers than a conventional RAG baseline, on datasets of around one million tokens.

The current GraphRAG documentation lists four query modes. Global search uses community summaries for questions about the whole corpus. Local search explores one entity and its neighbours. DRIFT search is local search with community context added. Basic search is ordinary vector retrieval, kept for questions it already handles well.

The cost problem and LazyGraphRAG

Building the graph and the summaries means many model calls up front. That is the main reason teams hesitate. Microsoft Research described LazyGraphRAG, which skips the upfront summarizing. Indexing uses noun phrase extraction to build a concept graph, and the model is used only at query time. Microsoft states that indexing cost is identical to vector RAG and 0.1% of the cost of full GraphRAG. It also states that answer quality on global queries is comparable to GraphRAG global search at more than 700 times lower query cost.

We would read those figures with care. They are the authors' own results on one dataset of 5,590 news articles with 100 synthetic questions, and the page offers no independent check. An editor's note on the page says the technology was integrated into Microsoft Discovery and, in public preview, Azure Local. The open-source GraphRAG documentation we opened on 2026-10-27 lists two indexing methods, Standard and Fast, and does not mention LazyGraphRAG, so we would not assume you can switch it on in that library.

When a graph is worth it

A 2026 systematic comparison under one evaluation protocol found that RAG and GraphRAG each have distinct strengths across tasks, and that neither wins everywhere. It also reported consistent gains from strategies that choose between the two or combine them. That matches the advice we give below.

Our engineering judgment is that graph retrieval is a specialist tool. Consider it when users ask about themes, trends or links across many documents, such as a research corpus, a case archive or an investigation file. Skip it when users ask lookup questions, where hybrid search with a reranker is cheaper and easier to explain. A graph also gives you something to maintain. Entities merge wrongly, and the graph needs a refresh when documents change.

Agentic RAG

This is the second branch of rung 6.

What it means

In agentic RAG, a model controls the retrieval loop instead of running one fixed search. The survey we read, revised in April 2026, describes it as an extension of RAG that places autonomous agents inside the retrieval pipeline, and names four patterns: reflection, planning, tool use and multi-agent collaboration. In plain terms, the model decides what to search for, reads what comes back, judges whether it is enough, and searches again with a better query if not.

A research example is Self-RAG. Its authors train one model to decide when to retrieve at all, instead of always retrieving a fixed number of passages, and to critique the passages and its own output using special reflection tokens.

When it helps and what it costs

Use it when one answer needs several lookups. "Compare the notice periods in our three supplier contracts" needs one search per contract, and a fixed single search will mix them up. A model that plans three searches handles it.

The cost is real. Anthropic's engineering post reports that agents typically use about 4 times the tokens of a chat, and multi-agent systems about 15 times. Those figures are for Anthropic's research system, not for every agentic RAG design. Our read is that the direction holds for any loop that searches more than once. Each extra search is another model call, so latency rises too.

The same post describes search subagents as intelligent filters that use search tools repeatedly to gather information, in contrast to static retrieval. The idea is the same as agentic RAG: the model, not a fixed pipeline, decides how much searching a question deserves.

Guard the loop

An agentic loop has the same dangers as any agent loop. It can repeat a query, drift, or spend without limit. Three guards, all in code and not in the prompt:

• a maximum number of searches per question
• a token or spend budget per question
• a stop when the model issues a query it has already issued

Our guide to the agent harness covers these limits in depth, with code we ran. Treat the retrieval loop as one more agent and give it the same guard rails.

We recommend adding agentic RAG last. A weak search inside an agent loop is a weak search that costs more.

Long context or RAG

Context windows are large, so a fair question is whether you need retrieval at all. The answer depends on size, cost and who is asking.

Anthropic's original contextual retrieval post advised that a knowledge base under 200,000 tokens, about 500 pages, can go straight in the prompt with no RAG. That was the standard window then. Current Claude models take up to 1M tokens, about 2,500 pages at the same ratio, though Claude Haiku 5.5 charges more for prompts over 100,000 tokens. Prompt caching makes a repeated prompt cheaper: a cache read costs 0.1 times the base input price for most models and 0.05 times for Claude Sonnet 5.5 and Opus 5.5. We use 200,000 tokens as our rule of thumb for "small", because beyond it you pay for, and the model must sift, far more text on every call.

The current Anthropic docs, read on 2026-10-27, say that more context is not automatically better: as token count grows, accuracy and recall degrade, an effect they call context rot. Position matters too. The foundational study, Liu et al., found that models often did best when the relevant passage was at the start or end of the input and worse when it sat in the middle. Two 2026 papers update that picture. A May 2026 audit of 11 long-context benchmarks tested nine models and found that some still lose a lot of accuracy when the task sits mid-context, while three of four newer releases stayed within 6 percentage points of their end-position accuracy at 64K tokens. An October 2026 study found that position bias is still noticeable in smaller models and in non-English languages, and that larger models and reasoning models reduce it sharply. Our read of the two papers together: the middle-of-context penalty has shrunk in current models but varies by model, so we recommend testing it on yours.

Our recommendation: do not choose one forever. A small, stable, shared handbook can go straight into a cached prompt. A large, changing, permissioned archive needs retrieval. Many teams end up with both.

Evaluation: recall@k and faithfulness

This is rung 1, and the one most projects skip. Without it, every other choice is a guess. Our LLM evaluation guide covers test sets and judges beyond retrieval.

Build the test set

A test set is a list of real questions, each paired with the passage or passages that answer it. We would start with fifty to a hundred for one corpus. This size is our rule of thumb, not a standard. Take the questions from real users, support tickets and search logs, not from the people who built the system. Include hard cases on purpose: exact codes, questions that need two documents, questions with no answer in the corpus, and questions where the old and the new policy disagree.

A subject expert has to mark the right passage for each question. This takes days, and in our judgment it is the best-spent time in the project.

Measure retrieval and the answer separately

The two parts fail in different ways, so measure each on its own.

The Ragas paper introduced metrics for exactly this split, and the current Ragas documentation still lists faithfulness, context recall, context precision and response relevancy among its retrieval and generation metrics. The Ragas paper assesses whether retrieval finds focused, relevant passages, whether the model uses them faithfully, and the quality of the generated text, without requiring human-labelled ground truth. Our read: reference-free scoring is useful for monitoring, but a model that grades a model can be wrong, so we check a sample by hand.

A worked example of the arithmetic

This example is illustrative arithmetic, not a measurement. Suppose your test set has 50 questions. For 46 of them a marked passage appears in the top 20. Recall@20 is 46 divided by 50, which is 0.92, and the failure rate is 8%. That is the same quantity Anthropic reports, so you can compare your 8% to its 5.7% baseline, with the caveat that your corpus and its are different.

Now you try rung 3 and add a keyword index. If 48 of 50 now appear in the top 20, recall@20 is 0.96 and the failure rate is 4%. You have halved the misses, and you know which rung did it. That is the point of climbing one rung at a time.

In our five-question toy run, recall@1 went from 0.60 for keyword search to 1.00 for hybrid. The number is real for that toy, and it says nothing about your documents.

Permissions on documents

A RAG system that searches everything and answers anyone is a data leak with a chat box on top. Our post on building an AI knowledge base explains why we sort out permissions before retrieval. We read OWASP's entry for this risk, LLM09:2026, Vector and Embedding Weaknesses, on 2026-10-27. It says to enforce tenant scoping inside the index query, not as a post-retrieval filter, and to validate it on the server. For high-sensitivity workloads it advises separate indexes per tenant or trust zone, and it says to apply access control at the chunk level, because a mostly public document can hold a confidential paragraph. It warns that in multi-tenant setups where similarity search runs across the full index before access control is applied, an attacker can infer that other tenants' documents exist from result counts, scores and timing. That is the same position we take: the filter belongs inside the query.

Filter inside the search

The rule is to apply the access check during retrieval. Store the access rule with each chunk, as a list of allowed groups or a tenant ID. At query time, pass the asker's identity and apply it as a filter, so disallowed chunks are never returned. Filtering the finished answer is too late. The model has already read the restricted text, and it can paraphrase it into an answer the filter does not recognize.

Keep the rules in step with the source

Permissions change. People leave, move teams and lose access to a folder. Three habits keep the index honest.

• Read permissions from the source system on every sync, not once at load.
• Remove a chunk from the index when its document is deleted or its access is withdrawn.
• Log which chunks were retrieved for each question, so you can answer "who saw what". OWASP recommends immutable logs of retrieval activity, including tenant scope, query, returned IDs and similarity scores. It also says stored embeddings can be inverted to recover source text, so a leaked vector store should be treated as a leak of the documents, and that embeddings should be deleted within a bounded time when the source document is deleted.

Test it with a canary

Plant a document that only one user may read, and include a unique phrase in it. Ask questions as other users that should surface it. The phrase must never appear in any answer. Run this test after every change to the index or to the permission sync. In our judgment it is the cheapest security test you can write.

OWASP also advises normalizing content before it is embedded, such as stripping zero-width characters and white-on-white text, tracking the provenance of every embedding, and applying human review to externally sourced content that goes into sensitive indexes. A document that anyone can edit is a way to put instructions into your model's context.

Freshness

Documents change, and an index that does not follow them gives confident, outdated answers. In the Stanford study above, we found the authors reporting that tools retrieved authority that no longer applied, such as a legal standard a court had overturned. The fixes below are our own engineering reasoning, not findings from a source.

• Update by change, not by rebuild. Hash each document. Re-chunk and re-embed only the ones whose hash changed.
• Delete properly. When a source document is removed, remove all its chunks. A deleted policy that stays searchable is a common failure.
• Carry dates. Put the document date and a version on every chunk, and rank newer versions higher when two chunks conflict.
• Mark superseded content. An old version should be tagged as replaced, not just buried.
• Watch the age. Track the share of answers that cite a chunk older than a threshold, and look at it monthly.

Contextual retrieval adds one more item. When a document changes, its context lines may change too, so regenerate them for the changed document. This is another reason to process documents one at a time with a warm cache.

Cost

RAG has an indexing cost, paid once per change, and a query cost, paid every time. The table separates what we read in sources from what we calculated.

The biggest recurring cost is usually the generation step, because the model reads about 16,000 tokens of passages for every question under those settings. Two levers help. We would send fewer chunks when your test set shows it does no harm, and use a smaller model for easy questions.

Staff time is the other cost, and it is larger than the model bill. Someone has to write and refresh the test set, review failures and own the index. Our enterprise RAG architecture post covers build budgets and upkeep. For the model side of the bill, see our post on how to reduce LLM costs without losing quality.

Failure modes to design for

RAG fails in repeating ways. The table lists ten. Each has a symptom you would see, a fix and a test you can run before launch. The last column says where the row comes from. "Our reasoning" means we derived it from how the parts work and no source states it.

Two notes on the table. First, the first four rows are the reason the ladder has the order it has. Each rung is the fix for a failure you can see in the test set. Second, the leak row deserves the most care in regulated work, because a retrieval bug becomes a disclosure.

A worked example from our own projects

The one RAG project we describe is the Manual Minds AI platform, and we report it only as its published case study states it. EdgeFirm built it for industrial equipment manuals. The case study states that each answer cites the document and page, that the PDF opens at that page, that text, tables and diagrams are all searched, and that each machine has its own knowledge base opened by scanning a QR code.

Reading the case study against the ladder, we see rung 2 as multimodal ingestion, where tables and diagrams are extracted and described, and every piece is stored with its file and page, and the permission idea as one isolated collection per machine, so a question about one machine cannot be answered from another's manual. The citation to a page is, in our reading, the metadata paying off at the end of the pipeline. The case study does not report recall figures, a reranker or contextual retrieval, so we make no claim that it used them, and none about its accuracy.

Here is how we would apply the ladder to a case of your own. Suppose you have 2,000 policy documents and a legal team that asks about notice periods.

1. Write 60 real questions from the legal team's inbox. Mark the answering clause for each. Include ten about superseded policies.
2. Split on clause headings and tag each chunk with document title, version, date and department.
3. Run vector search alone and record recall@20.
4. Add BM25 and RRF. Record again.
5. Add a reranker over the top 150. Record again.
6. Add context lines. Record again, and read the new misses by hand.
7. Stop climbing when the misses left are ones you can live with, or ones only a graph or an agent loop can fix.

The record at each step is the deliverable. It tells your leaders what each rung bought.

Common mistakes and when not to build RAG

Common mistakes

• Indexing everything on day one. More content lowers precision. Start with one corpus people ask about.
• Changing the model first. If the right passage is not retrieved, a better model cannot answer. Look at retrieval.
• Dense search only. It misses exact terms. Add BM25.
• Context lines on vectors only. Add them to the keyword index too.
• Filtering the answer for permissions. Filter inside the search.
• No test set. Without one you cannot tell a fix from a regression.
• Trusting a vendor's headline number. The 67% in this guide is Anthropic's, on its data, and we quote it as theirs. Run your own.
• Letting an agent search without limits. Cap searches and spend in code.
• Skipping freshness. A deleted policy that stays searchable will be quoted.
• Copying a chunk size from a blog. Test sizes on your own corpus.
• Treating the demo as the product. A prototype on clean files proves little; see proof of concept compared with production for what to build first.

When not to build RAG

• The knowledge base is small and stable, under about 200,000 tokens by our rule of thumb. Put it in a cached prompt.
• The questions need a calculation or a database query, not a passage. Use a tool that queries the data.
• The documents are too inconsistent to chunk or too poor to trust. Fix the documents first.
• Nobody will own the index, the test set and the permission sync. The system will rot.
• The answer must be exact and carry legal weight, and nobody will check it. The Stanford study we read reports RAG tools still giving wrong answers in legal research.

Production checklist

Use this before you launch. Every line should have a yes, or a written reason for no.

1. A test set of real questions, with marked passages, exists and runs in one command.
2. Recall@20 and failure rate are recorded for each rung you added.
3. Faithfulness is scored, and a person has read a sample of answers.
4. Chunks follow document structure, and tables are kept whole.
5. Every chunk carries source, page, date, version, document type and access rule.
6. Hybrid search is on, and exact-code questions are in the test set.
7. A reranker is on, or you have a recorded reason for no.
8. Context lines are added to both the vectors and the keyword index, or you have a recorded reason for no.
9. Permissions are applied inside the search, and the canary test passes.
10. Permission changes reach the index within a time you have agreed and written down.
11. Deleted documents lose their chunks.
12. Every answer shows its sources and says when it does not know.
13. Retrieval is logged per question, with chunk IDs.
14. Any agent loop has a search cap and a budget in code.
15. Cost per question is measured and has an alert.
16. A named person owns the index, the test set and the monthly review.

If you want help scoping a RAG system on your own documents, see our custom LLM and RAG development service. For a private deployment on your own documents, our post on a private AI chatbot for company documents describes how that works. If you are weighing RAG against tuning a model, read RAG, fine-tuning and prompt engineering.

What to ask a vendor

1. What do you retrieve for a question, and can I see it? A system that cannot show retrieved passages cannot be debugged.
2. How do you search? Ask whether it is keyword, vector or both, and whether a reranker is used.
3. How do you measure quality? Ask for recall and faithfulness on a test set from my data.
4. How do permissions work? Ask whether the filter runs inside the search and how fast changes arrive.
5. What happens when a document changes or is deleted?
6. What does one question cost, and what limits the cost?
7. Where does my data go? Ask which models and services see it.

In our judgment, a vendor with clear answers has built retrieval. A vendor who talks only about the model may not have.

Glossary