chunk-engine
Chunking Modes

Chunking Modes

Seven document modes plus dedicated spreadsheet modes — what each one emits, which parameter it reads, and how to pick.

View raw

The mode argument controls how a document is segmented. Document formats support seven modes; spreadsheet and delimited formats have their own set.

Showing examples for Python
— your choice follows you across the docs.

Document modes

ModeWhat it does
defaultEach format's natural element-level strategy (DOCX normalizes to structural).
structuralOne chunk per paragraph / heading / table / list.
sectionContent under a heading grouped into one chunk.
semanticAdjacent blocks merged while an eleven-signal ladder says they're the same idea.
sliding_windowOverlapping windows for dense retrieval.
sentenceA fixed number of sentences per chunk.
page_awareChunks aligned to the document's pages or slides.

Applies to DOCX (+ DOCM/DOTX/DOTM), DOC, PDF, PPTX (+ POTX/POTM/PPSX/PPSM), PPT, Markdown, HTML, TXT, MSG, EML/MBOX, ODT/ODP, JSON/JSONL/NDJSON, RTF, EPUB, and IPYNB.

A mode never gives you only its own content_type

mode="section" does not mean every chunk is content_type: "section". Most formats keep headings, tables and code blocks as their own typed chunks alongside the mode's chunks, and the exact set differs per format. Each mode page below states the set it really emits — read it before you write if chunk.content_type == mode.

Pick your mode

I want to…Best mode
Index every paragraph and heading individuallydefault / structural
Keep the body under a heading in one chunksection
Feed coherent passages to an LLM or embedding modelsemantic
Enforce a fixed sentence count per chunksentence
Build overlapping chunks for dense retrievalsliding_window
Cite a real page or slide numberpage_aware (on a format that has pages — see the table)

Default recommendation

Start with semantic when feeding an LLM and section when building a document search index. Use default/structural when you need fine-grained, element-level control.

Mode parameters

These are the four knobs the source-agnostic entry points (get_chunks / getChunks / chunks_rs::get_chunks) accept.

ParameterDefaultRead by
window_size3sliding_window (must be > 0)
overlap1sliding_window (must be < window_size)
sentences_per_chunk3sentenceand spreadsheets/CSV in row mode, see below
paragraphs_per_page15page_aware — units per page for formats with no page breaks; slides per chunk for PPTX; rows per chunk for CSV/TSV

Spreadsheet & delimited modes

Spreadsheets (XLSX / XLS / XLSM / XLSB / ODS / XLTX / XLTM) support six modes; CSV / TSV support three (plus default, an alias for row). Passing an unsupported mode raises immediately, and the engine names the set it accepts:

mode must be one of ['default', 'page_aware', 'row', 'semantic', 'sheet', 'sliding_window', 'table'] for XLSX, got: 'nope'
mode must be one of ['default', 'page_aware', 'row', 'sliding_window'] for CSV, got: 'nope'
ModeSpreadsheetsCSV / TSVcontent_type emitted
row (= default)yesyesrow_document (spreadsheet) / row_group (CSV)
tableyestable_region
sheetyessheet
sliding_windowyesyesrow_window
page_awareyesyessheet_region (spreadsheet) / row_group (CSV)
semanticyessemantic_group

table is not XLSX-only: it works for every extension in the spreadsheet family — .xlsx, .xls, .xlsm, .xlsb, .ods, .xltx, .xltm — all routed through the same calamine-backed chunker.

Rows per chunk through the unified entry point

There is no rows_per_chunk argument on get_chunks. The row count is derived from the two document parameters, and the derivation is not the same for spreadsheets and CSV:

Formatrow mode rows per chunkpage_aware rows per chunk
CSV / TSVmax(sentences_per_chunk, 1)3 at defaultsparagraphs_per_page15 at defaults
XLSX familysentences_per_chunk, except 31 (see below) → 1 at defaultswhole-sheet regions capped at 2,000 chars

`sentences_per_chunk == 3` is a sentinel for spreadsheets

For the spreadsheet extensions the value 3 — the API's own default — is read as "the caller did not ask" and mapped to rows_per_chunk = 1. Every other value passes through unchanged. The consequence: a deliberate sentences_per_chunk=3 is unreachable through get_chunks for a spreadsheet. You will get 1 row per chunk, not 3. If you genuinely want 3 rows per chunk, call the format-specific chunker (chunk_xlsx in Python, chunks_rs::formats::xlsx::chunk_with_options in Rust) and set rows_per_chunk directly.

CSV has no sentinel: sentences_per_chunk=3 really does mean 3 rows there.

Where the “10” comes from

You may have seen rows_per_chunk = 10 documented as the CSV default. That is the default of the format-specific chunk_csv() helper (and of ChunkOptions::default() in Rust) — it is not what the unified entry point uses. Through get_chunks / getChunks, CSV rows-per-chunk defaults to 3.

Other spreadsheet & delimited parameters

Reachable only through the format-specific chunkers, not get_chunks:

ParameterDefaultNotes
rows_per_chunk10 (chunk_csv / ChunkOptions)Rows per row chunk.
max_chunk_chars2000Cap for table / sheet / page_aware chunks (spreadsheets).
include_headerstrueRepeat the header row in each chunk.
sheet_namesallRestrict to specific worksheets.
skip_empty_rowstrueDrop blank rows.
serialize_askey_valueHow a row is rendered (spreadsheets).
delimiterauto-detected, \t ; | (CSV/TSV).
encodingautoauto / utf-8 / utf-8-bom / latin-1 / windows-1252 (CSV). auto tries UTF-8 strictly, then detects exactly as the .txt path does: BOMs, BOM-less UTF-16 from the NUL pattern, then a statistical charset detector for 8-bit encodings (so a windows-1251 or Big5 file decodes as itself, not as Windows-1252) — so a latin-1 CSV reads like a latin-1 .txt instead of raising. Naming an encoding explicitly still forces it. HTML needs no parameter: it reads its own <meta charset> (see Error Handling).

See the API Reference for the full signatures, and Metadata Reference for the keys each mode writes.

On this page