chunk-engine
Chunking Modes

section

Group the content under a heading into one chunk — and know where the heading itself ends up.

View raw

section groups the body under a heading into a single chunk, so a search hit returns a self-contained passage instead of one stray paragraph.

Showing examples for Python
— your choice follows you across the docs.

The heading is usually its OWN chunk

In every markdown-pipeline format — Markdown, HTML, TXT, PDF, EPUB, RTF, EML/MBOX, MSG, ODT, ODP, JSON, IPYNB — the heading is emitted as a separate chunk with content_type: "heading", and the section chunk that follows holds only the body. It is not folded in.

If you build "return the whole section" by taking chunks where content_type == "section", you drop every heading, and your index fills up with orphaned heading chunks that have no body. Reassemble instead: a heading chunk and the section chunk that follows it share the same heading_path.

DOCX, DOC, PPTX and PPT are the exception — their own chunkers put the heading line at the top of the section chunk's content.

Real input → real output

Run against a small notes.md with three headings, at default parameters:

from py_chunks import get_chunks

# One chunk per section body, with the heading trail in heading_path.
# The heading itself stays a SEPARATE chunk — it is not folded into the body.
for c in get_chunks("notes.md", mode="section"):
    print(c["content_type"], "|", c["metadata"]["heading_path"])

# heading | ['Chunking Notes']
# section | ['Chunking Notes']
# heading | ['Chunking Notes', 'Why structure matters']
# section | ['Chunking Notes', 'Why structure matters']
# heading | ['Chunking Notes', 'Modes']
# section | ['Chunking Notes', 'Modes']

Six chunks from three sections: heading, body, heading, body, heading, body. The heading_path is what pairs them back up.

What it emits

Formatcontent_type values emitted
Markdown-family — Markdown, HTML, TXT, PDF, EPUB, RTF, EML/MBOX, MSG, ODT, ODP, JSON, IPYNBheading and section
DOCX, DOC, PPTX family, PPTsection only — heading text is the first line of the chunk
Spreadsheets, CSV/TSVnot supported (mode must be one of [page_aware, row, semantic, sheet, sliding_window, table] for XLSX)

A document with no headings produces no heading chunks, obviously — and JSON/JSONL render no headings of their own, so they usually have none.

Measured: prose_heavy.md → 10 section + 10 heading; arxiv_1301.3781_word2vec.pdf → 31 section + 10 heading; all_round.docx → 9 section, 0 heading.

Units and formats

FamilyUnit grouped into a section
Markdown-family (md, html, txt, pdf, epub, rtf, eml/mbox, msg, odt, odp, json, ipynb)markdown block — paragraphs, lists, tables and code blocks under one heading
DOCX, DOCparagraph, from one heading to the next
PPTXslides grouped by PPTX sections, or a title-divider heuristic
PPTslide

Parameters

section reads none of the four mode parameters. Boundaries come entirely from the document's heading structure.

2,000-character cap

A section body longer than 2,000 characters is split at a paragraph boundary into several section chunks. The pieces carry split_part and split_total so you can rejoin them; a section that fits in one chunk leaves both null. This applies to the markdown-family formats — DOCX and PPTX have their own section builders.

Metadata

section chunks carry section_heading, heading_path, section_level, block_types, char_count, paragraph_count and chunk_index — the exact set varies by format. Full listing in the Metadata Reference.

When to use it

  • Document search indexes where a hit should return a whole passage.
  • RAG where you want enough surrounding context to answer, scoped to one topic.
  • Navigation / outline features keyed on headings — the standalone heading chunks are exactly the outline.

On this page