Introduction
One document chunking engine for RAG — 36 formats, one Rust core, with byte-identical bindings for Python, JavaScript, and Rust.
chunk-engine turns any document into clean, retrieval-ready chunks in a single call. A Rust core does the parsing and segmentation; thin bindings give you the same API — and the same output — from Python, JavaScript, or Rust, across 36 file formats.
quickstart.py
from py_chunks import get_chunks, stream_chunks, get_markdown
# Batch — works for every supported format
chunks = get_chunks("document.pdf")
chunks = get_chunks("notes.md", mode="semantic")
chunks = get_chunks("deck.pptx", mode="sliding_window", window_size=3, overlap=1)
for chunk in chunks:
print(chunk["content"], chunk["content_type"], chunk["metadata"])
# Streaming — constant memory over huge files
for chunk in stream_chunks("large.pdf", mode="section"):
handle(chunk)
# Markdown conversion
md = get_markdown("report.docx")Every chunk has three fields — content, a typed content_type, and
metadata — ready to embed, index, or feed to an LLM.
One engine, three languages
| Package | Install | Runtime |
|---|---|---|
| py-chunks (PyPI) | pip install py-chunks | Python, via PyO3 |
| js-chunks (npm) | npm install js-chunks | Node · Bun · Deno · browsers, via WASM |
| rs-chunks (crates.io) | cargo add rs-chunks | Rust — the reference engine |
Byte-identical output
The SDKs wrap the same engine and are parity-checked to emit exactly the same chunks — 2204/2214 comparisons identical, with the remainder tracing to the reference engine's own non-determinism. See Languages & parity.
Why chunk-engine
- Rust-backed speed. Parsing and chunking run in a compiled core, not a stack of interpreted dependencies.
- One API, every format. The same entry points work for Word, PowerPoint, Excel, PDF, HTML, Markdown, email, eBooks, notebooks, and more.
- Structure-aware chunking. Seven document modes and dedicated spreadsheet modes keep headings, tables, and lists intact.
- Streaming built in. Yield chunks with constant memory over huge files.