# Introduction

One document chunking engine for RAG — 36 formats, one Rust core, with byte-identical bindings for Python, JavaScript, and Rust.

**chunk-engine** turns any document into clean, retrieval-ready chunks in a
single call. A Rust core does the parsing and segmentation; thin bindings give
you the same API — and the **same output** — from **Python, JavaScript, or
Rust**, across **36 file formats**.

```python
from py_chunks import get_chunks, stream_chunks, get_markdown

# Batch — works for every supported format
chunks = get_chunks("document.pdf")
chunks = get_chunks("notes.md",  mode="semantic")
chunks = get_chunks("deck.pptx", mode="sliding_window", window_size=3, overlap=1)

for chunk in chunks:
    print(chunk["content"], chunk["content_type"], chunk["metadata"])

# Streaming — lazy iteration; genuinely incremental for pdf and xlsx
for chunk in stream_chunks("large.pdf", mode="section"):
    handle(chunk)

# Markdown conversion
md = get_markdown("report.docx")
```

```ts
import { getChunks, streamChunks, getMarkdown } from "js-chunks";

// Batch — works for every supported format
let chunks = await getChunks("./document.pdf");
chunks = await getChunks("./notes.md",  { mode: "semantic" });
chunks = await getChunks("./deck.pptx", { mode: "sliding_window", windowSize: 3, overlap: 1 });

for (const c of chunks) {
  console.log(c.content, c.contentType, c.metadata);
}

// Streaming — async-iteration ergonomics over the same results
for await (const chunk of streamChunks("./large.pdf", { mode: "section" })) {
  handle(chunk);
}

// Markdown conversion
const md = await getMarkdown("./report.docx");
```

```rust
use chunks_rs::{formats::csv, get_chunks, get_markdown};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Batch — dispatch by extension. Positional args:
    // get_chunks(path, mode, window_size, overlap, sentences_per_chunk,
    //            paragraphs_per_page)
    let chunks = get_chunks("document.pdf", "default", 3, 1, 3, 15)?;
    println!("{} chunks", chunks.len());

    let chunks = get_chunks("notes.md", "semantic", 3, 1, 3, 15)?;

    for c in &chunks {
        println!("[{}] {}", c.content_type, c.content);
        // c.metadata is a serde_json::Value with format-specific provenance
    }

    // Streaming is per-format, and yields the same chunks one at a time
    for c in csv::stream("data.csv", "row", 10, 5, 1, true, None, "utf-8", true)? {
        let c = c?;
        println!("{}", c.content_type);
    }

    // One-shot Markdown conversion
    let md = get_markdown("report.docx")?;
    println!("{}", md.len());
    Ok(())
}
```

## What a chunk looks like

Every chunk has exactly three fields: `content`, a typed `content_type`, and a
`metadata` object. Here is a complete one — no elisions — from `notes.md` in
`semantic` mode at default parameters:

```python
chunk = get_chunks("notes.md", mode="semantic")[3]

# Every chunk is a dict with exactly three keys:
{
    "content": "A naive character splitter cuts mid-sentence and mid-table. The retrieved\npassage then answers half a question, and the model fills in the rest. Keeping\na table whole costs nothing at index time and saves a wrong answer at query\ntime.",
    "content_type": "semantic",
    "metadata": {
        "avg_block_length": 234,
        "block_types": ["paragraph"],
        "chunk_index": 3,
        "document_metadata": {"source_type": "md", "total_input_blocks": 7},
        "has_list": False,
        "heading_path": ["Chunking Notes", "Why structure matters"],
        "keyword_density": 0.6,
        "merge_reasons": [],
        "paragraph_count": 1,
        "primary_merge_reason": "initial",
        "section_heading": "Why structure matters",
        "section_level": 2,
    },
}
```

```ts
const chunk = (await getChunks("./notes.md", { mode: "semantic" }))[3];

// Same values as py-chunks; content_type is surfaced as contentType, and
// metadata keys stay exactly as the engine emits them (snake_case).
{
  content: "A naive character splitter cuts mid-sentence and mid-table. The retrieved\npassage then answers half a question, and the model fills in the rest. Keeping\na table whole costs nothing at index time and saves a wrong answer at query\ntime.",
  contentType: "semantic",
  metadata: {
    avg_block_length: 234,
    block_types: ["paragraph"],
    chunk_index: 3,
    document_metadata: { source_type: "md", total_input_blocks: 7 },
    has_list: false,
    heading_path: ["Chunking Notes", "Why structure matters"],
    keyword_density: 0.6,
    merge_reasons: [],
    paragraph_count: 1,
    primary_merge_reason: "initial",
    section_heading: "Why structure matters",
    section_level: 2,
  },
}
```

```rust
use chunks_rs::get_chunks;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let chunks = get_chunks("notes.md", "semantic", 3, 1, 3, 15)?;
    // Chunk is { content: String, content_type: String, metadata: serde_json::Value }
    // and derives Serialize, so this prints the shape the other two SDKs return.
    println!("{}", serde_json::to_string_pretty(&chunks[3])?);
    Ok(())
}

// {
//   "content": "A naive character splitter cuts mid-sentence and mid-table. The retrieved\npassage then answers half a question, and the model fills in the rest. Keeping\na table whole costs nothing at index time and saves a wrong answer at query\ntime.",
//   "content_type": "semantic",
//   "metadata": {
//     "avg_block_length": 234,
//     "block_types": [
//       "paragraph"
//     ],
//     "chunk_index": 3,
//     "document_metadata": {
//       "source_type": "md",
//       "total_input_blocks": 7
//     },
//     "has_list": false,
//     "heading_path": [
//       "Chunking Notes",
//       "Why structure matters"
//     ],
//     "keyword_density": 0.6,
//     "merge_reasons": [],
//     "paragraph_count": 1,
//     "primary_merge_reason": "initial",
//     "section_heading": "Why structure matters",
//     "section_level": 2
//   }
// }
```

`content` goes to your embedding model; `content_type` tells you what kind of
block it was; `metadata` carries the provenance you need to filter, rank and
cite. The full catalogue is in [Output Schema](/docs/output-schema) and
[Metadata Reference](/docs/metadata-reference).

## One engine, three languages

- **Python** (PyPI): `pip install py-chunks`
- **JavaScript** (npm): `npm install js-chunks`
- **Rust** (crates.io): `cargo add rs-chunks`

| Package | Import name | Runtime |
| --- | --- | --- |
| **py-chunks** (PyPI) | `py_chunks` | Python 3.9+, via PyO3 |
| **js-chunks** (npm) | `js-chunks` | Node · Bun · Deno · browsers, via WASM |
| **rs-chunks** (crates.io) | `chunks_rs` | Rust — the reference engine |

  The SDKs wrap the same engine and are parity-checked to emit exactly the same
  chunks: **3,222 / 3,222 comparisons byte-identical (100%)** across every
  fixture × every mode, re-verified 2026-08-08. The one known divergence is a
  scanned PDF that WASM cannot rasterise. See [Languages &
  parity](/docs/languages) for the numbers, the caveats, and the commands that
  reproduce them.

## Why chunk-engine

- **Rust-backed speed.** Parsing and chunking run in a compiled core, not a stack
  of interpreted dependencies.
- **One API, every format.** The same entry points work for Word, PowerPoint,
  Excel, PDF, HTML, Markdown, email, eBooks, notebooks, and more.
- **Structure-aware chunking.** Seven document modes and dedicated spreadsheet
  modes keep headings, tables, and lists intact.
- **Streaming built in.** Consume chunks one at a time instead of waiting for a
  whole document. Genuinely incremental — bounded memory — for PDF, CSV and
  spreadsheets in Rust (spreadsheets only in `row` and `sliding_window` modes),
  and for PDF and spreadsheets in Python; lazy delivery over a completed parse
  everywhere else. In JavaScript it is always lazy delivery over a completed
  parse, never bounded-memory. [The honest matrix](/docs/streaming) says which
  is which, per format and per SDK.

## Start here

  - [Installation](/docs/installation) — Install for Python, JavaScript, or Rust.
  - [Quick Start](/docs/quick-start) — Chunk your first document in a few lines.
  - [Playground](/playground) — Chunk a document in the browser — no install.
  - [Supported formats](/docs/supported-formats) — All 36 extensions and what each one produces.
  - [Languages & parity](/docs/languages) — Choose an SDK, and how the three stay identical.
  - [Benchmarks](/benchmarks) — Extraction coverage vs Docling and Unstructured.
  - [API Reference](/docs/api-reference) — Signatures for all three languages.