One chunking engine.
Python, JavaScript, Rust.
A fast, high-fidelity document chunking engine for RAG. 36 formats, one Rust core — with byte-identical bindings for Python, JavaScript, and Rust.
pip install py-chunkspy-chunks · js-chunks · rs-chunks
from py_chunks import get_chunks
# One API. Every format. Rust underneath.
chunks = get_chunks("report.pdf", mode="semantic")
for chunk in chunks:
print(chunk["content_type"], "->", chunk["content"][:48])
# heading -> Q3 Financial Summary
# semantic -> Revenue grew 18% quarter-over-quarter, driven...
# table -> | Region | Revenue | YoY |Coverage
Milliseconds, not seconds
A Rust core does the parsing and segmentation. On a neutral x86 box, over the files all three tools could read, chunk-engine is 84-121x faster than Docling and Unstructured by median time per file — and reads formats they skip.
Across a full 654-file corpus it read 634 of 637 legitimate documents and every one of 31 format directories, against 78% and 68% for peers running their full documented install. The three it missed are named on the benchmark page.
See the benchmarkWhy chunking
Chunking is the hidden lever in retrieval quality
Retrieval is only as good as its chunks. A boundary drawn at a character count cuts through sentences and tables; chunk-engine draws them on the document's own structure instead.
Naive fixed-size splitting
Sentences cut mid-word, tables severed, headings orphaned: what a boundary drawn at a fixed character count does to a document.
chunk-engine — structure-aware
Every chunk arrives with a typed content_type, and boundaries follow the document's own structure — so a table or a list survives as one unit instead of being cut at a character count.
Illustration, not measured output — the panels above are drawn to show the mechanism. For measured figures see the benchmarks.
Formats
36 formats, one engine
Office, PDF, web, plain-text, data, email, eBooks, and notebooks — grouped by family. Every one of them ships in all three SDKs.
Word
5OOXML + legacy binary Word
PowerPoint
6OOXML + legacy binary PowerPoint
Spreadsheets
7Excel + OpenDocument sheets
Text + page-scoped images
Web & Markup
4Markup and rich text
OpenDocument
2LibreOffice / OpenOffice documents
Plain & Data
6Text and structured data
Outlook + MIME email
eBooks & Notebooks
2EPUB and Jupyter
Every format listed here ships in every SDK — py-chunks, js-chunks and rs-chunks read all 36 from the same engine, with byte-identical output.
See it
Watch a document become chunks
A sample document on the left; colored chunk blocks on the right, keyed by content type. Toggle the mode and watch the blocks regroup.
Q3 Financial Summary
Revenue grew 18% quarter-over-quar…
Regional Breakdown
Regional performance diverged shar…
Growth bullet list
Region / Revenue table
Ingest Example
get_chunks(…) call
One chunk per element — each heading, table, and list stands alone.
One engine, three languages
Same chunks, everywhere
Pick the SDK that fits your stack — Python, JavaScript, or Rust. They wrap the same engine and produce byte-identical output.
Byte-identical, verified
All three SDKs wrap the same Rust engine, so they emit exactly the same chunks. Parity is checked over every fixture × every mode, by harnesses you can run yourself — and the one remaining difference is a scanned PDF that WASM cannot rasterise.
pip install py-chunks- 4,748 / 4,748 chunk comparisons byte-identical (100%)
- 374 / 375 image-extraction fixtures identical
- 23 / 24 PDF fixtures identical (chunks + markdown + images)
Modes
Seven ways to chunk a document
default, structural, section, semantic, sliding_window, sentence, page_aware — plus row / table / sheet for spreadsheets.
defaultElement-level chunks — one per paragraph or heading.
Best for: Fine-grained, element-level control.
structuralIndex every paragraph and heading individually.
Best for: Fine-grained retrieval over document structure.
sectionrecommendedKeep all content under a heading together in one chunk.
Best for: Section-level search / document indexes.
semanticrecommendedGroup semantically coherent passages together.
Best for: Feeding an LLM or embedding model.
sliding_windowOverlapping windows with configurable size and overlap.
Best for: Dense retrieval / sliding-context inference.
sentenceEnforce a fixed number of sentences per chunk.
Best for: Tight token budgets.
page_awarePreserve the document's original page layout.
Best for: Page-referenced citations.
Spreadsheets (XLSX / XLS / ODS) add dedicated modes — CSV and TSV take row and sliding_window, but not table or sheet:
Quick start
Chunk your first document
One call. Switch the language tab to see the exact API for your SDK.
from py_chunks import get_chunks, stream_chunks, get_markdown
# Batch — works for every supported format
chunks = get_chunks("document.pdf")
chunks = get_chunks("notes.md", mode="semantic")
chunks = get_chunks("deck.pptx", mode="sliding_window", window_size=3, overlap=1)
for chunk in chunks:
print(chunk["content"], chunk["content_type"], chunk["metadata"])
# Streaming — lazy iteration; genuinely incremental for pdf and xlsx
for chunk in stream_chunks("large.pdf", mode="section"):
handle(chunk)
# Markdown conversion
md = get_markdown("report.docx")Start chunking in one line
Install for your language, point it at a file, get clean chunks back. No services, no config, no heavy dependency tree.
pip install py-chunks