Blog

Chunk a PDF on its headings, not its characters

Fixed-size chunking does not cut words in half. It does something worse, and quietly. Here is the real failure, and eighty lines of Python that avoid it.

By Harish9 September 20267 min read

Fixed-size chunking is the default in every tutorial, and the usual criticism of it is wrong. A decent length-based splitter does not slice words in half, because it looks for a paragraph break, then a line break, then a sentence break before it gives up and cuts. The real failure is quieter: it separates every heading from the section it introduces, so the chunk that contains the words “Hedging policy” contains none of the hedging policy.

Why length is the default

Not because anyone thinks it is good. Because it is the only thing available when the extraction step handed you a wall of text.

If your converter flattened the headings, there is no structure left to cut on, so the splitter counts to a thousand characters and cuts there. Every downstream decision inherits that. This is worth saying plainly because it changes what you should fix: chunking strategy is usually not the problem, it is the symptom. If the extraction lost the structure, no splitter can invent it back.

The rest of this post assumes you have Markdown with the headings intact. If you do not, get that first.

The real failure, on a real document

Here is a short treasury report converted to Markdown, then split with a recursive character splitter at a 400 character target. This is actual output, not an illustration.

The first chunk:

# Currency Exposure Review

This review covers the twelve months to 31 March. It was prepared by the treasury
team and reviewed by the audit committee before publication.

## Hedging policy

The second:

The group hedges between 60 and 80 per cent of forecast transactional exposure on a
rolling twelve month basis. Instruments are limited to forwards and vanilla options.
Speculative positions are prohibited under the treasury mandate.

## Sensitivity

Look at where the headings landed. Every one of them sits at the end of the chunk before the section it belongs to. Not one word was cut in half. Not one sentence was broken. The splitter did its job correctly and the result is still wrong.

Two things follow, and both are bad.

The heading is in the wrong chunk. Ask “what is the hedging policy?” and the strongest keyword match is chunk one, which contains the phrase and none of the answer. The actual policy is in chunk two, which never says the words “hedging policy” at all.

The content lost its label. Chunk two is three sentences about hedging ratios that never name the company, the period, or the fact that this is a policy rather than a description of what happened. On its own it is close to unfindable, because a question asked in the reader’s words has very little to match against.

Now the same document, cut on its headings instead:

Currency Exposure Review > Hedging policy

The group hedges between 60 and 80 per cent of forecast transactional exposure on a
rolling twelve month basis. Instruments are limited to forwards and vanilla options.
Speculative positions are prohibited under the treasury mandate.

The heading is attached to its own content, and the chunk states what it is.

Four things a chunk needs

A boundary a human chose. A heading is where an author decided one idea stopped and another began. Nothing you compute will beat that, and it is already in the file.
Its own context, in its text. The chunk should say what it is about even when read alone, because that is exactly how it will be read.
A record of where it came from. Page numbers, so an answer built on it can be checked.
A size that fits your budget. Last, not first. Size is a constraint to respect, not a strategy to organise around.

The code

Eighty lines of Python, no framework. Three functions: parse the Markdown into sections, split anything oversized, assemble the chunks.

It is deliberately not built on a framework, because the frameworks move faster than a blog post can and because there is not enough here to justify a dependency. The header-aware splitters that ship with the common libraries do the first half of this well and none of them carry page numbers through, which is the half that decides whether your answers can be cited.

First, the parser. It walks the Markdown line by line, tracking the heading stack and the current page.

import re

HEADING = re.compile(r"^(#{1,6})\s+(.*\S)\s*$")
PAGE = re.compile(r"^\s*<!--\s*page\s+(\d+)\s*-->\s*$")
FENCE = re.compile(r"^\s*(```|~~~)")


def sections(markdown, start_page=1, max_depth=3):
    """Yield (heading_path, [(line, page), ...]) for each section."""
    stack, pending, page, fence = [], [], start_page, None
    for line in markdown.split("\n"):
        fenced = FENCE.match(line)
        if fenced or fence:
            if fenced:
                fence = fenced.group(1) if fence is None else (
                    None if fenced.group(1) == fence else fence)
            pending.append((line, page))
            continue
        marker = PAGE.match(line)
        if marker:
            page = int(marker.group(1))
            continue
        head = HEADING.match(line)
        if head and len(head.group(1)) <= max_depth:
            if pending:
                yield [t for _, t in stack], pending
            level = len(head.group(1))
            stack = [e for e in stack if e[0] < level] + [(level, head.group(2))]
            pending = []
            continue
        pending.append((line, page))
    if pending:
        yield [t for _, t in stack], pending

Three details in there are worth more than they look.

The heading stack is tagged with levels, not stored as a plain list. A document that jumps from # straight to ### leaves a gap, and with a plain list the next ### nests underneath its own sibling instead of replacing it. Filtering on level handles the jump correctly.

Fenced blocks are tracked, because a # inside a code block is a comment and not a heading. Our own converter never emits fences, but you will point this at other people’s Markdown eventually.

A heading with no body yields nothing and still contributes to the path of what follows. A chunk containing only a heading retrieves nothing and takes up a slot.

Next, the splitter, for sections that are too long to keep whole. This is the standard separator ladder, which is what makes the comparison at the top of this post a fair one rather than a strawman.

SEPARATORS = ["\n\n", "\n", ". ", " "]


def split_spans(text, limit):
    """Offsets of pieces at most `limit` chars, cut at the best separator."""
    out, pos = [], 0
    while pos < len(text):
        if len(text) - pos <= limit:
            out.append((pos, len(text)))
            break
        window, cut = text[pos:pos + limit], -1
        for sep in SEPARATORS:
            at = window.rfind(sep)
            if at > 0:
                cut = at + len(sep)
                break
        cut = cut if cut > 0 else limit
        out.append((pos, pos + cut))
        pos += cut
    return out

It returns offsets rather than strings, which matters for the next part: page numbers are attached to positions, so slicing the text first would throw away the ability to say which page a piece came from.

Finally, assembly.

def chunk(markdown, max_tokens=512, start_page=1):
    out = []
    for path, lines in sections(markdown, start_page):
        while lines and not lines[0][0].strip():
            lines.pop(0)
        while lines and not lines[-1][0].strip():
            lines.pop()
        if not lines:
            continue

        body, spans = "", []
        for text, page in lines:
            start = len(body) + 1 if body else 0
            body = body + "\n" + text if body else text
            spans.append((start, len(body), page))

        limit = max_tokens * 4
        pieces = split_spans(body, limit) if len(body) > limit else [(0, len(body))]
        for a, b in pieces:
            while a < b and body[a].isspace():
                a += 1
            while b > a and body[b - 1].isspace():
                b -= 1
            if a >= b:
                continue
            pages = [p for s, e, p in spans if e > a and s < b]
            prefix = " > ".join(path)
            out.append({
                "text": f"{prefix}\n\n{body[a:b]}" if prefix else body[a:b],
                "heading_path": path,
                "first_page": pages[0],
                "last_page": pages[-1],
            })
    return out

The blank line trimming at the top of the loop looks like tidiness and is not. Page markers sit on their own lines, so a section that ends with a blank line before the next page’s marker will otherwise pick up the following page number and report a range one page too wide. That was a real bug, found by a test that checked a page number rather than checking that the text looked right.

Run it on the report from earlier and you get four chunks, each carrying its heading path and the page it came from, with the deepest one labelled Currency Exposure Review > Sensitivity > Counterparty limits.

Put the path in the text, not the metadata

This is the single change with the best ratio of effort to effect, and it is counter-intuitive enough that plenty of production systems get it wrong.

Storing the heading path as a metadata field feels right. It is structured, it is queryable, it keeps the text clean. It also does nothing for retrieval, because the embedding model reads the text and never sees your metadata. A chunk whose vector is computed from three sentences about hedging ratios is findable by questions about hedging ratios and nothing else. The same chunk with Currency Exposure Review > Hedging policy as its first line is findable by a question about the company’s policy, which is the question a person actually asks.

Keep the metadata too. Use it for filtering. Just do not mistake it for something the model can read.

This is the cheap version of a broader technique. Writing a sentence or two of real situating context per chunk with a language model does better still, and it costs more than nothing, which is a subject in its own right.

Keep the page numbers, or the answer cannot be checked

The page markers this code reads are the  comments our converter emits. They are HTML comments specifically so that they are invisible when the Markdown is rendered and readable when it is parsed.

The consequence is worth being blunt about. If you convert with page markers turned off, or with the horizontal rule style that carries no number, the page information is gone and nothing downstream can recover it. Your system can then produce an answer but cannot tell anyone where it came from, and an answer nobody can check is an answer nobody is allowed to act on. In a legal, clinical or financial setting that is not a missing feature, it is the whole difference between a demo and a deployment.

If you converted a page range rather than the whole document, pass its first page as start_page. The converter writes a marker between pages and not before the first one, so the opening run of text has no marker of its own and the code assumes page one.

Size it with the right tokeniser

max_tokens above is multiplied by four to get a character budget, which is a rough approximation and is labelled as one. If you need the real number, do not reach for tiktoken: it is a different model family’s tokeniser and it will not give you correct counts for a model it was not built for. Most providers expose a token counting endpoint, it is free to call, and it is the only way to get an exact figure.

The approximation is fine for deciding where to cut. It is not fine for budgeting a context window, which is a different job.

What this deliberately does not do

It does not merge small sections. A heading followed by one line produces a small chunk, and merging small neighbours would produce fewer, better-sized chunks. We left it out because a merged chunk has to claim a heading path, and every rule for choosing which one is wrong some of the time. A slightly small chunk that is honestly labelled beats a well-sized chunk labelled with a section it is only half from. If you add merging, merge siblings only and take the shared parent path.

It does not handle tables. A table flattened into text has already lost the link between a cell and its column header before this code sees it, and no chunking strategy recovers that. If your documents are mostly tables, this is not your bottleneck.

It does not do overlap. Overlap is a workaround for cutting in the wrong places. Cut in the right places and you need much less of it. Add it to the length-based path if you want it; it does not belong on a heading boundary.

Try it on your own document

Convert a PDF with our PDF to Markdown converter, leave the page markers on the default comment setting, save the output, and run the code above over it. The conversion happens in your browser, so you can do this with a document you are not allowed to upload anywhere, which tends to be the same set of documents worth building retrieval over in the first place.

If the output looks wrong, check the conversion before you check the chunker. The four things that go missing during extraction all show up as strange chunks, and it is worth ruling out a document that is a photograph of paper before anything else.

The short version

Length-based chunking does not cut words in half. It orphans every heading from its section, which is worse because it looks fine.
Headings are boundaries a human already chose. Use them, and fall back to length only when a single section is too long.
Put the heading path in the chunk text. Metadata is invisible to the embedding model.
Carry the page numbers through, or your answers cannot be cited.
Chunking strategy is downstream of extraction quality. Fix the conversion first.