Benchmarks
Measured, not claimed
A full-corpus competitive run — 446 real files across all 36 formats. chunk-engine is a document-understanding engine, so it's measured against its real peers, Docling and Unstructured (text-splitters can't read a file at all).
- Corpus
- 446 real files · all 36 formats · 3 runs/file (median)
- Machine
- Apple M1 Max · 10 cores (8P+2E) · 32 GB
- OS
- macOS 26.5.1
- Runtimes
- py-chunks 0.5.0 · docling 2.115.0 · unstructured 0.24.1 · markitdown 0.1.5
Apple-Silicon numbers — compare tools within this run, not across machines. A neutral x86 cloud run is a planned follow-up.
Coverage
Reads more formats than anything else
One representative real fixture per extension, every tool attempted for real — nothing pre-skipped.
Text-splitters (LangChain, semchunk, Chonkie, semantic-text-splitter) read no files directly — they always chunk markitdown's extracted text.
Structure integrity
Keeps oversized units whole
Feed each tool tables, lists, and code blocks larger than any chunk budget. Only one tool refuses to cut them in half.
| Tool | Table | List | Code |
|---|---|---|---|
| chunk-engine · section | 50% | 100% | 100% |
| chunk-engine · default | 20% | 100% | 100% |
| chunk-engine · semantic | 20% | 100% | 100% |
| Docling | 0% | 0% | 0% |
| Unstructured | 0% | 0% | 0% |
| LangChain | 0% | 0% | 0% |
| semchunk · Chonkie · s-t-s | 0% | 0% | 0% |
Kept-whole rate: how often an oversized table, list, or code block lands in one chunk instead of being split. Every competitor tested — including Docling and Unstructured — split 100% of them. chunk-engine keeps oversized lists and code whole by design, and its section mode keeps half of oversized tables whole too.
Speed
Milliseconds, not seconds
Native Rust parsing with no ML model loading and no per-element Python overhead. Pooled over all 446 files.
Higher is better. Docling and Unstructured also errored on ~half the corpus, so their median is measured over an easier subset — generous to them, not harsh.
Speed by format
Fastest in every category
Per-category multiples are more credible — and more dramatic — than one pooled number.
| Category | chunk-engine | Docling | Unstructured | markitdown+lc | Fastest by |
|---|---|---|---|---|---|
| 61.38 | 11,351 | 1,970 | 1,353 | 185× vs Docling | |
| PowerPoint | 1.17 | 694.43 | 1,861 | 436.96 | 1,591× vs Unstructured |
| Word | 0.44 | 183.11 | 134.04 | 73.89 | 416× vs Docling |
| Ebook | 5.39 | 359.94 | 1,922 | 118.43 | 357× vs Unstructured |
| Spreadsheet | 0.49 | 12.01 | 25.61 | 19.01 | 52× vs Unstructured |
Median ms per file (lower is better). chunk-engine is fastest in every category measured — margins widen wherever a competitor's pipeline is heaviest.
Accuracy
Precise typed chunks
Is the content_type label actually right? Ground truth extracted directly from source markup, independent of every tool.
| Precision | chunk-engine | Docling | Unstructured |
|---|---|---|---|
| table | 1.00 | 0.854 | 1.00 |
| code | 1.00 | 0.476 | 1.00 |
Perfect precision on tables and code. Docling over-labels code blocks (0.48 precision — many non-code chunks tagged “code”). On headings and lists chunk-engine favours fewer, higher-value typed units — it under-labels at the anchor level but retains ~99% of the content.
Metadata
Richer provenance per chunk
chunk-engine trails on raw page-number population, but attaches a format-specific metadata schema that no competitor exposes.
Page-number populationpage-aware formats
chunk-engine trails on raw page-number population — a fair loss, offset by the schemas on the right.
Per-format metadata schemas
Docling and Unstructured expose only page_number. chunk-engine attaches a schema per extension:
Prose quality
Doesn't fragment sentences either
The one axis text-splitters compete on. Verified by hand across every mode.
Coverage
Formats × modes
Which chunking modes each format family supports — derived directly from the library source.
| Format family | default | structural | section | semantic | sliding_window | sentence | page_aware | row | table | sheet |
|---|---|---|---|---|---|---|---|---|---|---|
| Word (.docx .doc …) | ||||||||||
| PowerPoint (.pptx .ppt …) | ||||||||||
| PDF (.pdf) | ||||||||||
| Web & Markup (.html .md) | ||||||||||
| Plain text (.txt) | ||||||||||
| Email (.msg .eml) | ||||||||||
| OpenDocument (.odt .odp) | ||||||||||
| eBooks / Notebooks (.epub .ipynb) | ||||||||||
| JSON (.json .jsonl) | ||||||||||
| Rich text (.rtf) | ||||||||||
| Spreadsheets (.xlsx .xls .ods …) | ||||||||||
| Delimited (.csv .tsv) |
Scorecard
The honest tally
Every axis measured — including the ones chunk-engine loses. Credibility comes from the whole picture.
| Axis | Winner | Margin |
|---|---|---|
| Format coverage | chunk-engine | 35/36 vs 18–26; text-splitters 0/36 |
| Structure integrity (oversized units) | chunk-engine | only tool that keeps any whole — competitors 0% |
| Content-type precision (table / code) | chunk-engine | 1.0 / 1.0 vs 0.14–0.85 |
| Content-type recall (heading / list) | Docling | chunk-engine retains ~99% of content but under-labels at anchor level |
| Metadata field richness | chunk-engine | only tool with per-format schemas |
| Page-number population | Unstructured / Docling | 0.67–0.77 vs chunk-engine 0.585 |
| Reading order | tie | all four ~0.92–0.93 (within measurement noise) |
| Prose sentence-integrity | tie | 0 splits any mode; sliding_window at parity |
| Speed (pooled, 446 files) | chunk-engine | 55–221× faster; fastest in every category |
Full disclosure
What we didn't hide
The same report surfaces every defect it found. Credibility comes from disclosing them, not burying them.
- 35/36 formats, not 36: XLSB workbooks containing a chart sheet fail to parse (a calamine reader bug).
- HIGH severity, found by this benchmark: literal “&” is silently dropped from DOCX/PPTX text (“R&D” → “R D”) — a fix is pending, not yet released.
- Page-number metadata population (0.585) trails Docling/Unstructured (0.67–0.77) on page-aware formats — offset by far richer per-format metadata schemas.
- On headings/lists chunk-engine under-labels at anchor granularity (but retains ~99% of the content) — it favours higher-value typed units.
- All numbers are Apple Silicon (M1 Max) and not cross-machine comparable; a neutral x86 cloud run is planned.
Every number here is from a single real run on the environment above — no sampling, no cherry-picking. Absolute figures are Apple-Silicon specific; a neutral x86 cloud run is a planned follow-up.