chunk-engine
Chunking Modes

Chunking Modes

Seven document modes plus dedicated spreadsheet modes — how to pick the right one.

View raw

The mode argument controls how a document is segmented. Document formats support seven modes; spreadsheet and delimited formats have their own set.

Document modes

ModeWhat it does
defaultElement-level chunks (per format; DOCX normalizes to structural).
structuralOne chunk per paragraph / heading / table / list.
sectionAll content under a heading grouped into one chunk.
semanticSemantically coherent passages grouped together.
sliding_windowOverlapping windows for dense retrieval.
sentenceA fixed number of sentences per chunk.
page_awarePreserves the document's page (or slide) layout.

Applies to DOCX, DOC, PDF, PPTX, PPT, Markdown, HTML, TXT, MSG, EML/MBOX, ODT/ODP, JSON, RTF, EPUB, and IPYNB.

Pick your mode

I want to…Best mode
Index every paragraph and heading individuallydefault / structural
Keep all content under a heading in one chunksection
Feed coherent passages to an LLM or embedding modelsemantic
Enforce a fixed sentence count per chunksentence
Build overlapping chunks for dense retrievalsliding_window
Preserve page layout for page-referenced citationspage_aware

Default recommendation

Start with semantic when feeding an LLM and section when building a document search index. Use default/structural when you need fine-grained, element-level control.

Mode parameters

ParameterDefaultApplies to
window_size3sliding_window (must be > 0)
overlap1sliding_window (must be < window_size)
sentences_per_chunk3sentence (must be > 0)
paragraphs_per_page15page_aware (PPTX defaults to 5, meaning slides per chunk)

Spreadsheet & delimited modes

Spreadsheets (XLSX / XLS / XLSM / XLSB / ODS / XLTX / XLTM) support six modes — row, table, sheet, sliding_window, page_aware, semantic — and CSV / TSV support three: row (default), sliding_window, page_aware.

ModeWhat it does
rowN data rows per chunk (content_type row_document / row_group).
tableOne chunk per detected table region (XLSX/XLS only).
sheetOne chunk per worksheet.
sliding_windowOverlapping row windows (row_window).
page_awareRows grouped into page-sized chunks (sheet_region).
semanticSemantically grouped rows (semantic_group).

Spreadsheet & delimited parameters

ParameterDefaultNotes
rows_per_chunk1 (XLSX) / 10 (CSV)Rows per row chunk.
max_chunk_chars2000Cap for table / sheet / page_aware chunks (XLSX).
include_headerstrueRepeat the header row in each chunk.
sheet_namesallRestrict to specific worksheets (XLSX).
skip_empty_rowstrueDrop blank rows.
serialize_askey_valueHow a row is rendered (XLSX).
delimiterauto, \t ; | (CSV/TSV).
encodingutf-8utf-8 / utf-8-bom / latin-1 / windows-1252 (CSV).

Through the unified get_chunks / getChunks, spreadsheets use row when you pass the default mode, and rows-per-chunk is derived from sentences_per_chunk / sentencesPerChunk. For direct control of rows_per_chunk, delimiter, max_chunk_chars, or sheet_names, call the format-specific chunker (chunk_xlsx / chunk_csv in Python, chunks_rs::formats::{xlsx,csv} in Rust) — see the API Reference.

On this page