chunk-engine
Chunking Modes

sentence

A fixed number of sentences per chunk — for the blocks that are made of sentences.

View raw

sentence splits prose into groups of sentences_per_chunk sentences. When you have a tight token budget and want predictable chunk sizes, this gives you consistent, sentence-aligned boundaries.

Showing examples for Python
— your choice follows you across the docs.

Real input → real output

from py_chunks import get_chunks

# Groups of sentences_per_chunk sentences (default 3), never crossing a block.
# Headings and tables stay whole and carry no sentence count.
for c in get_chunks("notes.md", mode="sentence"):
    print(c["content_type"], "|", c["metadata"].get("actual_sentence_count"))

# heading | None
# sentence | 2
# heading | None
# sentence | 2
# table | None
# heading | None
# sentence | 3
# sentence | 1     <- the short final group of a 4-sentence paragraph

Note the last group: a four-sentence paragraph at sentences_per_chunk=3 produces a chunk of 3 and a chunk of 1. Groups never span a block, so a short tail is normal.

Most chunks in this mode are not sentences

sentence mode does not turn the whole document into sentence groups. In the markdown-family formats it splits paragraphs and leaves every other kind of block alone: headings, tables, code blocks and lists pass through as their own typed chunks.

Lists are deliberately not split — cutting a bullet list at "three sentences" produces fragments that mean nothing. A bullet_list chunk in this mode carries actual_sentence_count: 0, which is the tell: it was never sentence-split.

What it emits

Formatcontent_type values emitted
Markdown, HTML, TXT, PDF, EPUB, RTF, EML/MBOX, MSG, ODT, ODP, JSONsentence, plus heading, bullet_list, code_block and table wherever the document has them
IPYNBcode_block for code cells; heading / bullet_list / sentence for markdown cells
DOCX, DOC, PPTX family, PPTsentence only
Spreadsheets, CSV/TSVnot supported

Measured: code_heavy.md → 13 sentence + 22 heading + 17 code_block + 2 table; _stress_big_list.md → 63 bullet_list + 1 sentence + 1 heading; all_round.docx → 11 sentence and nothing else.

If your pipeline assumes content_type == "sentence", a list-heavy Markdown document will silently drop almost all of its content.

Units and formats

FamilyWhat gets sentence-split
Markdown-familyparagraph blocks only. Headings, lists, tables and code blocks pass through untouched.
DOCX, DOCevery paragraph
PPTX, PPTall slide text, split across slides
Spreadsheets, CSV/TSVnot supported

Parameters

ParameterDefaultNotes
sentences_per_chunk3Sentences per group. Must be > 0, or sentences_per_chunk must be greater than 0 is raised.

The same argument means something else for spreadsheets

sentences_per_chunk is also what the unified entry point uses to derive rows-per-chunk for spreadsheets and CSV — including a sentinel that makes a deliberate 3 unreachable. See Chunking Modes → rows per chunk.

Metadata

sentence chunks carry actual_sentence_count (the number this chunk really holds — the last group of a block is often short), sentences_per_chunk (what you asked for), source_paragraph_index, heading_path, section_heading, section_level and chunk_index. Pass-through chunks (heading, table, code_block) omit the sentence keys entirely; bullet_list chunks keep them with actual_sentence_count: 0. Full listing in the Metadata Reference.

When to use it

  • Strict token budgets where uniform chunk sizes matter.
  • Models or indexes that assume short, even inputs.
  • Sentence-level highlighting or citation.

On this page