Architecture
A thin language binding dispatches by file type into one compiled Rust engine.
chunk-engine is one Rust engine (rs-chunks) with thin language bindings. Each
binding handles ergonomics — input detection, dispatch, and result shaping —
while all parsing and chunking happens in Rust.
- Python (
py-chunks) binds the engine natively via PyO3. - JavaScript (
js-chunks) loads the engine compiled to WASM (Node, Bun, Deno, browsers). - Rust (
rs-chunks) is the engine itself — the reference implementation.
Because every binding calls the same core, the output is byte-identical (see Languages & parity).
Language binding — Python · JavaScript · Rust
get_chunks / getChunks · stream · get_markdown · from path / bytes / fileobj / upload / URL
↓ detect source · resolve extension
Dispatcher
extension → format chunker (docx, pdf, pptx, xlsx, csv, html, md, …)
↓ call the compiled Rust core (PyO3 · WASM · native)
Rust engine
parse → classify blocks (ContentType) → apply mode → build chunks + metadata
↑ typed chunks — list[dict] · Chunk[] · Vec<Chunk>
How a call flows
- Input detection. The unified functions inspect
source— path, bytes, file-like, framework upload, orhttp(s)URL — and normalize it. In Python, non-path inputs are written to a temporary file with the correct extension and cleaned up afterward; the Rust and WASM builds parse bytes in memory viafrom_bytesentry points. - Dispatch. The file extension selects a format chunker. The extension is
the single source of truth, which is why bytes inputs require a
filename. - Rust engine. The compiled core parses the document, classifies each block
into a
ContentType, applies the requestedmode, and returns typed chunks with metadata. - Result shaping. The binding maps the Rust result into the language's
native shape — a Python
list[dict], a JSChunk[], or a RustVec<Chunk>— plus the image variants when requested.
Design principles
- Rust does the work. Parsing and chunking are compiled, not interpreted, and don't pull in a heavy dependency tree in any language.
- One dispatch table. Every format flows through the same entry points, so the API surface stays small as formats are added.
- Streaming built in. The engine can yield chunks incrementally, exposed as
stream_chunks(Python),streamChunks(JavaScript), and per-formatstream(...)(Rust). Memory is bounded for the truly incremental formats (PDF, CSV, and the Markdown/HTML/TXT state machines); container formats parse once, then emit lazily — see Streaming for the per-format profile. - Byte-identical across bindings. Because every binding calls the same core, the output matches exactly — verified by the parity suite (see Languages & parity).