Blog

Give every chunk its context before you embed it

A chunk saying "the ratio fell to 1.4" is unfindable. Contextual retrieval fixes it, and prompt caching is the difference between cheap and abandoned.

By Harish9 min read

Splitting a document destroys the context that made each piece findable. A chunk that reads “the ratio fell to 1.4 in the second half” does not say which company, which year, or which ratio, so no question a person would actually ask will retrieve it. The fix is to write that context back into the chunk before embedding, and the reason most people try it once and give up is that the obvious implementation costs about ten times what it should.

The chunk that cannot be found

Here is a chunk from a treasury report, exactly as a splitter would produce it.

A ten per cent weakening of sterling against the euro would reduce reported
operating profit by approximately 4.2 million pounds before hedging, and by 1.1
million pounds after the hedges described above are taken into account.

Read it cold. Whose profit? Which year? What hedges, described above where? The chunk is a perfectly good paragraph and a nearly useless retrieval unit, because everything that would let you find it is in the sentences that are no longer next to it.

The cheap partial fix is the heading path, which the chunking post covers: prepend Currency Exposure Review > Sensitivity and the chunk is now findable by a question about sensitivity. That is real and it costs nothing.

It is also not enough. The heading path is the author’s label, not a description of this particular chunk. It will not tell you that the entity is the group rather than a subsidiary, that the period is the twelve months to 31 March, or that “the hedges described above” refers to a 60 to 80 per cent rolling policy in a different section.

Two fixes, with different shapes

Contextual retrieval asks a language model to write one or two sentences situating each chunk inside its document, and prepends that to the chunk before embedding. Anthropic’s write-up of the technique, and independent implementation reports including one from AWS, put the improvement in retrieval precision in the range of roughly 5 to 15 per cent depending on the corpus. Those are not our numbers and we have not reproduced them; treat them as the reason to try it rather than as a promise.

Late chunking attacks the same problem in vector space instead of in the text. It embeds the whole document at the token level first, then pools the token vectors into chunk vectors, so each chunk’s vector was computed while the rest of the document was still visible. The original paper reports the largest gains on documents full of pronouns and back-references, which is exactly the case that breaks naive splitting.

They are not alternatives so much as different costs. Late chunking needs an embedding model that supports it and no extra inference calls. Contextual retrieval works with any embedding model and costs one language model call per chunk. Most people reach for the second, because it does not constrain the rest of the stack, and that is the one worth writing about because its cost is the thing everybody gets wrong.

Why the obvious implementation gets abandoned

One call per chunk, each one carrying the whole document so the model can see what it is situating.

Take a 50 page report. Using the common rule of thumb of roughly 800 tokens per page, that is about 40,000 tokens of document, and say it produces 80 chunks. The naive loop sends the document 80 times:

80 chunks x 40,000 tokens = 3,200,000 input tokens

At the published rate for a small fast model, that is a few pounds for one document. It is not ruinous, and that is exactly the trap: it is cheap enough to run once on a demo and far too expensive to run across a corpus. So people try it on one file, see the retrieval improve, look at what it would cost for forty thousand files, and quietly drop the idea.

Prompt caching, and the one rule that makes it work

The document is identical on every one of those 80 calls. Only the chunk changes. That is precisely the shape prompt caching exists for.

The rule that decides whether it works is that caching is a prefix match. The request is assembled in a fixed order, tools first, then the system prompt, then the messages, and the cache key is the exact bytes up to the breakpoint you mark. Anything that varies has to sit after everything that does not.

So the document goes in the system block with the cache breakpoint on it, and the chunk goes in the user message, after it. Put them the other way round and you have written 80 different prefixes and cached nothing.

The economics, using the published multipliers: a cache write costs about 1.25 times normal input price for the five minute cache, and a read costs about a tenth. So the first call pays a small premium and the other 79 pay a tenth.

1 write   40,000 x 1.25       =    50,000 token-equivalents
79 reads  79 x 40,000 x 0.1   =   316,000 token-equivalents
                              =   366,000 total

against   80 x 40,000         = 3,200,000 uncached

Just under nine times less, for a change of request shape rather than a change of technique. Those figures are arithmetic on published per-token rates rather than a measurement of ours, and the per-page assumption is a rule of thumb. Run it on your own document and read the real numbers off the response, which the code below does.

The code

import anthropic
from concurrent.futures import ThreadPoolExecutor

client = anthropic.Anthropic()
MODEL = "claude-haiku-4-5"

INSTRUCTION = """Here is a chunk taken from the document above.

<chunk>
{chunk}
</chunk>

Write one or two sentences situating this chunk within the document, so it can be
understood and found on its own. Name the document, the section it belongs to, and
any entity, period, currency or unit the chunk refers to without naming. Reply with
the situating context only."""


def describe(document, chunk_text):
    return client.messages.create(
        model=MODEL,
        max_tokens=300,
        system=[
            {
                "type": "text",
                "text": f"<document>\n{document}\n</document>",
                "cache_control": {"type": "ephemeral", "ttl": "1h"},
            }
        ],
        messages=[{"role": "user", "content": INSTRUCTION.format(chunk=chunk_text)}],
    )


def contextualise(document, chunks, workers=8):
    # The first call writes the cache. Nothing can read an entry that is still
    # being written, so fanning out immediately means every call pays full price.
    first = describe(document, chunks[0]["text"])
    rest = []
    if len(chunks) > 1:
        with ThreadPoolExecutor(max_workers=workers) as pool:
            rest = list(pool.map(lambda c: describe(document, c["text"]), chunks[1:]))

    out = []
    for chunk, message in zip(chunks, [first, *rest]):
        context = message.content[0].text.strip()
        out.append({**chunk, "text": f"{context}\n\n{chunk['text']}"})
    return out

The document sits in system with the breakpoint on it. The chunk sits in messages, which renders after the system block. max_tokens is capped low on purpose: the output should be one or two sentences, and a cap is cheaper than a prompt begging for brevity.

Three things that silently cost you the saving

Every one of these fails without an error. You get a working pipeline and a bill that is eight times larger than you think it is, which is why the verification step at the end of this post is not optional.

1. Anything that varies above the breakpoint. A timestamp in the system prompt, a request id, a user name, a dictionary serialised without sorted keys, a system section that is only included on some calls. Each of these changes the prefix bytes and invalidates everything after them. The rule is easy to state and easy to break by accident: the stable content goes first, and nothing above the last breakpoint may differ between calls.

2. Fanning out the first call. This is the one that catches careful people, because it is the result of optimising. A cache entry cannot be read until the response that writes it has begun coming back. Launch all 80 requests at once and none of them can read what the others are still writing, so all 80 pay full price. You have parallelised correctly and saved nothing. Send one, wait for it, then fan out, which is what the code above does.

3. The minimum cacheable prefix, which is not the same on every model. Below a model-specific threshold, nothing caches at all and there is no warning: you simply see a cache creation count of zero. The thresholds vary by roughly a factor of eight across current models, and the smaller, cheaper models tend to have the higher minimums, which is the opposite of what you would guess when you picked a cheap model for this job.

The practical consequence is a real limit on the technique. Contextual retrieval with caching is very cheap on long documents and genuinely not cheap on short ones. A five page memo may sit under the threshold, cache nothing, and cost you the full naive price per chunk. That is usually tolerable in absolute terms, because a short document is cheap to send anyway, but it means the saving you measured on a 200 page report will not appear on a folder of two page letters. Check the current minimum for the model you are using before you assume the arithmetic above applies.

There is a fourth thing that is not silent but is easy to forget: the default cache lifetime is five minutes. If contextualising a document takes longer than that, the entry expires part-way through and you start paying to write it again. The code above asks for the one hour lifetime, which costs more to write and is trivially worth it once there are more than a couple of reads.

Verify it, do not assume it

The response reports what actually happened. Read it.

u = message.usage
print(u.input_tokens, u.cache_creation_input_tokens, u.cache_read_input_tokens)

On the first call you should see a large creation count and a read count of zero. On every call after it, the reverse. If the read count stays at zero across calls that should be sharing a prefix, one of the three problems above is present, and no amount of reasoning about your code will tell you which: diff the exact bytes of two requests and look for the thing that moved.

This takes one line and it is the difference between knowing the technique is paying for itself and believing it is.

Which model, and why not the best one

Writing a sentence that says “this is from the sensitivity section of the currency review, covering the group’s exposure for the twelve months to 31 March” is not a reasoning task. It is summarising text that is directly in front of the model.

Use the cheapest, fastest model you have. The output quality difference on this particular job is small and the price difference is several times, multiplied by one call per chunk across your whole corpus. This is one of the few places in a retrieval stack where the cheap model is not a compromise.

The interaction with the caching minimum above is the one wrinkle: the cheap models sometimes have the higher cache thresholds, so on short documents the cheap model may cache nothing while a more capable one would have. Measure before assuming.

When not to bother

When your chunks are already self-describing. Reference documentation, product catalogues and anything where each section names its own subject already carry their context. The heading path alone will do.

When the corpus is small enough to read whole. If every document fits in the model’s context window and you have few enough of them, retrieval is solving a problem you do not have.

Before you have fixed extraction. Contextualising a chunk that was produced from interleaved columns just means paying a model to write a confident description of nonsense. The four things that go missing during extraction all come before this, and this technique will not rescue any of them.

That last one is the general rule for this whole cluster: everything downstream is built on the text, and no amount of clever retrieval recovers information that was destroyed before it was indexed. If you want to see what your own documents look like coming out of a layout-aware conversion, our PDF to Markdown converter runs in the browser, so you can check a document you are not free to upload before committing to a pipeline around it.

The short version

  • Splitting a document strips each piece of the context that made it findable.
  • Writing one or two situating sentences back into each chunk before embedding fixes that. Published implementations report retrieval precision gains of roughly 5 to 15 per cent.
  • The naive implementation sends the whole document once per chunk and is why people abandon the idea.
  • Prompt caching cuts that by close to an order of magnitude, but only if the document sits above the breakpoint and the varying chunk sits below it.
  • Three things silently destroy the saving: anything varying above the breakpoint, fanning out the very first call, and documents shorter than the model’s minimum cacheable prefix.
  • Read the cache counters off the response. One line, and it is the only way to know.

Further reading