# Changelog Release history for py-chunks, js-chunks and rs-chunks — one engine, three packages, one version number since 0.6.0. Release history for all three packages. The newest entry at the top of this page is the newest release; for what a registry is serving right now, check [PyPI](https://pypi.org/project/py-chunks/), [npm](https://www.npmjs.com/package/js-chunks) or [crates.io](https://crates.io/crates/rs-chunks) — a publish can lag a release. **From 0.6.0 the three packages share one version number.** They are one engine with three bindings and are released together, so `py-chunks`, `js-chunks` and `rs-chunks` now move in lockstep — `js-chunks` and `rs-chunks` jump from `0.1.0` to `0.6.0` to join `py-chunks`, not because 5 releases happened in between. History before 0.6.0 tracks **py-chunks**, the longest-running package, from which the shared engine grew. Entries below are reconstructed from git tags and commit messages; `py-chunks` and `js-chunks` also ship their own `CHANGELOG` files from 0.6.1. ## 0.6.4 *Released 2026-09-12.* Token-budget features in the Python and JavaScript bindings — no engine behaviour changes, so cross-SDK output is untouched. - **`fit_tokens` / `fitTokens` now hold their budget on every input.** The split ladder bottoms out at a character-level bisection sized by your own counter, so whitespace-free text — CJK prose, long URLs, base64, minified code — can no longer be silently emitted over budget as one indivisible run. Parts reassemble losslessly. - **Pass the tokenizer you're holding.** `counter` accepts a tiktoken `Encoding`, a HuggingFace tokenizer, an encoding or model name string (`"cl100k_base"`, `"gpt-4"`, a HF id), or a plain `str -> int` callable. Name resolution imports tiktoken/transformers lazily; HuggingFace counters default to `add_special_tokens=False`. Exported as `py_chunks.coerce_counter` too. - **`fit_tokens(..., overlap=)`** — an `int` token count or a fraction of the budget, shared between the parts of a chunk that had to be split. Scoped within one engine chunk on purpose (overlapping across chunks would smear metadata and structural boundaries), and overlapped parts still respect the budget — asserted by test, not assumed. - **`chunk_text(text)` / `chunkText(text)`** — chunk a bare string through the plain-text pipeline, no file or extension. Composes with `fit_tokens` / `fitTokens`. - The JavaScript `fitTokens` gains the same character-level floor and the same `overlap` option (an int, or a fraction of the budget), with the same ≤ budget enforcement. Counter *name* resolution stays Python-only — JS has no canonical tokenizer registry, so `fitTokens` keeps its callable contract. ## 0.6.3 *Released 2026-09-01.* Twenty correctness fixes across seventeen formats, from a per-format review of the engine against each format's specification. Every fix ships with a test that fails without it. Highlights: - **Silent data loss closed**: a spreadsheet header heuristic that deleted every row above a deep "header" row (one sheet lost 329 of 331 rows with no error); `.ppt` emitting slides deleted by earlier saves as live content (`total_slides` 305 for a 14-slide deck); CSV/TSV records beginning `#` silently dropped; `.docx` tabs and symbol-font glyphs producing no output; notebook image payloads in Jupyter's split-line form dropped; GeoJSON-style envelope keys deleted around the record array. - **Corruption closed**: RTF no longer synthesises a character from two unrelated unpaired surrogates; non-UTF-8 HTML and text now decode via a statistical charset detector instead of a blanket Windows-1252 fallback (a windows-1251 page went from 927 mojibake runs to fully recovered); JSON object members keep document order; PDF honours `/CropBox` so out-of-page pre-press bands are no longer extracted (clipping only under an author-declared crop); mailbox splitting no longer turns a body line starting with `From ` into a phantom message; `text/css` resources in HTML email are no longer inlined as prose; a comment inside a hyperlink in `.odt` no longer crashes the caller. - `.msg` recipient types are read at their specified offset, so `Cc:` lines reappear; EPUB image order is deterministic; slide-number/date placeholder caches in PowerPoint are no longer emitted as slide text. - `pip install py-chunks` 0.6.2 could abort the host process on a malformed `.xls`; 0.6.3 returns a catchable error. ## 0.6.2 *Released 2026-08-08.* Argument validation, error-contract completeness and a Python 3.9 that actually imports — no new formats or modes, and no change to the chunks any already-valid call returns. **py-chunks** - **Python 3.9 actually works.** `import py_chunks` raised `TypeError: unsupported operand type(s) for |` on CPython 3.9: 41 runtime-evaluated PEP 604 unions in function signatures across `_sources.py` and four chunker modules. `from __future__ import annotations` is now present in every package module, so annotations are never evaluated at definition time. The declared `requires-python = ">=3.9"` is a true statement again. - `py_chunks.__version__`, read from the installed distribution's metadata, so it cannot drift from what pip installed. Exported through `__all__`. - A Python 3.9 wheel-test job gates publishing, alongside the existing 3.11 job — the 3.11 job proves the stable ABI spans versions, the 3.9 job proves the Python sources contain no post-3.9 syntax. - **Three streaming helpers skipped argument validation.** `stream_chunk_doc()` and `stream_chunk_ppt()` ran no mode or range check at all, so a mis-typed `mode` silently streamed **default-mode** chunks instead of raising; `stream_chunk_docx()` raised `NotImplementedError: Streaming for mode coming soon`, which was doubly wrong — nothing was unimplemented, all seven DOCX modes stream. All three now run the same validators as their batch counterparts and raise `ValueError` at the call, before the first item is produced. **`stream_chunk_docx` no longer raises `NotImplementedError`**; if you caught it, catch `ValueError`. The other 16 `stream_chunk_*` helpers were audited and were already clean. - **Range messages now match the engine's wording.** `chunkers/doc.py`, `chunkers/ppt.py` and `chunkers/docx.py` said `window_size must be > 0` (and the same shape for `sentences_per_chunk` / `paragraphs_per_page`), while the spreadsheet family said `window_size must be >= 1`. Everything now says the engine's `must be greater than 0`, so the sentence is identical across formats *and* across the three SDKs. The exception *type* is unchanged — keep branching on that, not on the text. **js-chunks** - **`ChunkError` now covers the wrapper's own argument validation.** Three host-side checks — a filesystem path used off Node, an unsupported `source` type, and byte input with no filename — previously threw a plain `Error` or `TypeError`. They now throw `ChunkError` with `kind: "invalid-arg"`. Message text is unchanged, so only the thrown *type* differs; code that caught `TypeError` for the unsupported-source case must catch `ChunkError` instead. - **A missing or unreadable path is now a `ChunkError` too.** `fs.readFileSync` sat outside the wrapping, so `getChunks("./missing.md")` escaped as a raw Node `Error` — the single most common failure fell outside a contract documented as complete. It now throws `ChunkError` with `kind: "io"`, preserving Node's message (`ENOENT: no such file or directory, open './missing.md'`). With this and the three checks above, **every** JavaScript failure is a `ChunkError`. Code that tested `e.code === "ENOENT"` on a non-`ChunkError` branch should move to `e instanceof ChunkError && e.kind === "io"`. - **`overlap >= window_size` is now `invalid-arg` everywhere.** It was `kind: "parse"` for the markdown-family formats and `"invalid-arg"` for DOCX/PPTX/XLSX, because `md`/`txt`/`html` lifted their builder result straight into `Parse` and `csv::chunk_from_bytes` skipped its range checks entirely. A shared `validate_mode_args` now runs before dispatch, and the CSV checks were added. The same applies to `window_size = 0`, `sentences_per_chunk = 0` and `paragraphs_per_page = 0`. `kind` is now a reliable argument/parse split — see [Error Handling](/docs/error-handling). With the EPUB and spreadsheet fixes below, "everywhere" is now literally true. - **Four escapes from the `ChunkError` contract, all through `opts`.** `getChunks(bytes, null)` and `getMarkdown(bytes, null)` threw a raw `TypeError`; a **throwing getter** on `opts` propagated as whatever it threw, and `streamChunks` hit it too via `{ ...opts }` even though it reads no key itself; and a `Blob` whose `arrayBuffer()` rejects escaped as its own rejection. A single `snapshotOpts` now runs at all five entry points — so a getter fires at most once, and never mid-parse — and the `Blob` read is wrapped, reporting `kind: "io"`, the same kind the path branch reports for the same condition. `null` is accepted as "no options". - **Numeric options are validated, not coerced.** The wasm boundary takes `usize`, so `windowSize: -1` used to return chunks, `2.5` truncated to 2 and `"3"` coerced to 3; a non-string `filename` or `mode` surfaced as `kind: "unknown"` with the message `memory access out of bounds`. All of these are now `kind: "invalid-arg"` with an actionable message. Out-of-range values reuse the engine's snake_case sentence (`window_size must be greater than 0`); type mistakes name the camelCase key you typed. **Zero is deliberately still handled by the engine**, so its message names the parameter the target format really uses — `paragraphsPerPage: 0` on a CSV reports `rows_per_chunk must be greater than 0`. `overlap: 0` remains legal. - **XML entity references in OOXML relationship targets are now decoded.** Hyperlink URLs in `get_markdown` output carried literal `&` because relationship attributes were read without entity folding. Markdown output only; chunk content was never affected. - `streamChunks` is documented honestly in the README as an ergonomic wrapper, not incremental streaming. No API change; the TSDoc already said so. **rs-chunks** — and therefore all three SDKs - **EPUB now validates its mode arguments.** EPUB does not route through the markdown pipeline, so it never reached the shared `validate_mode_args`, and a bad argument produced an **empty chunk list** instead of an error — `get_chunks("book.epub", mode="sliding_window", window_size=100, overlap=100)` returned `[]`. This was true in **all three** SDKs: py-chunks checked the `mode` string in its own layer and nothing else, so the numeric arguments were unvalidated there too. A new `epub::validate_args` now runs at all four entry points, after the extension check and before the book is loaded, so the bytes route rejects without parsing. Valid calls are unchanged (Moby Dick still yields 1,440 chunks at the defaults). **If you relied on an empty array for bad EPUB arguments, you now get `ValueError` / `ChunkError` / `ChunkError::InvalidArg`.** - **The spreadsheet family rejects `paragraphs_per_page = 0` in `page_aware`.** XLSX/ODS/XLSB/XLSM paginate by whole-sheet regions and never read the value, so zero was silently accepted while every other format rejected it. The check is **scoped to `page_aware`** — the only mode that reads the parameter — because an unconditional check would reject a value that no other format rejects outside its mode. - **One sentence for a range rejection.** The five spreadsheet sites that said `window_size must be >= 1` now say the engine's `must be greater than 0`, matching the document formats and the Python layer. No exception type or `kind` changed. ## 0.6.1 *Released 2026-08-08.* Packaging, typing, error ergonomics and semver hardening — no new formats or modes. **js-chunks** - **Typed errors.** Every engine failure now throws a `ChunkError` — a real `Error` subclass — with a `kind` of `unsupported`, `invalid-arg`, `parse`, `io` or `unknown`. Failures used to be thrown as bare strings, so `e instanceof Error` was false and there was no stack; both work now. `ChunkError` and the `ChunkErrorKind` type are exported. Messages are unchanged. See [Error Handling](/docs/error-handling). - **A supported bundler path.** New `js-chunks/web` subpath export (plus `js-chunks/web/chunks_wasm_bg.wasm`) for vite/webpack apps; the default entry still auto-loads on Node, Bun, Deno and unbundled browser ESM. See [Installation](/docs/installation#bundlers-vitewebpack). - `chunkPdfMarkdownWithImages` and `normalizePdfMarkdown` are now public API, joining `chunkPdfMarkdown` for host-parsed PDF workflows. - `"sideEffects": false` for tree-shaking. The package is ESM-only — `require()` needs Node ≥ 20.19. **py-chunks** - **Stable-ABI wheels.** One binary wheel per platform now covers CPython 3.9+; previously only 3.13 got a wheel and everyone else fell back to a source build. Linux aarch64 wheels added. - **The package is typed** — `py.typed`, complete stubs for the compiled module, and the `Typing :: Typed` classifier, so mypy and pyright see full annotations. - **In-memory input no longer touches disk.** Bytes, file objects, uploads and URLs are passed straight to the engine instead of round-tripping through a temp file (output verified byte-identical). - The GIL is released while the engine parses and chunks, so a long parse no longer blocks other Python threads. **rs-chunks** - **`ChunkError` is now `#[non_exhaustive]`.** A `match` on it in a downstream crate must carry a wildcard arm; this lets the engine add variants without a semver-major bump, and it is a **breaking change for 0.6.0 code that matched exhaustively**. See the [Rust API reference](/docs/api-reference/rust#chunkerror-and-result). - **Every dispatch entry point is panic-guarded.** All six public functions run the parse behind `catch_unwind`, so a panic in the engine or in a third-party parser returns `ChunkError::Parse("internal parser panic: …")` instead of unwinding into the caller. An adversarial-input test suite came with it. - `ChunkMode` implements `FromStr`; the inherent `from_str` stays as an `Option`-returning convenience. - The temp-file fallback for unsupported bytes was removed — the bytes API is now genuinely filesystem-free. - Internal only: the docx and pptx monoliths were split into 24 focused submodules, and `.doc` opens its CFB container once over borrowed bytes instead of three times over full copies. Behaviour-identical — golden snapshot 0 changed, parity 3,222/0. ## 0.6.0 *Released 2026-08-07.* The largest correctness release so far: **64 engine changes** closing every defect in the tracker. Chunk *content* changes for most formats, almost always because more text is now extracted or because it is laid out correctly. **Re-index after upgrading.** Extraction improved across PDF, `.doc`, `.txt`, `.md`, email and OpenDocument, so chunk text and boundaries differ from 0.5.0. Embeddings generated with an earlier version will not match text produced by this one. **Behaviour changes worth knowing before you upgrade** - **Oversized blocks are now bounded in every mode.** A table, list or code block larger than the chunk budget used to come back as a single chunk; it is now split at row, item or line boundaries — never mid-row. A single 11k-token chunk broke the size contract downstream embedding models depend on. - **Image names are identical across the three SDKs.** They were not before: the content hash included a native-width length prefix, so JavaScript produced different names from Python and Rust for every format. JS names change with this release. - **`py-chunks` has no runtime dependencies.** `pypdfium2` is gone from the install; PDFium is vendored in the wheel. - Error types and messages were made consistent across bindings and modes, so code matching on exact strings may need updating. **PDF** — the parser is now this project's own (`lopdf` object model, our layout engine, WebAssembly-clean) rather than a third-party pipeline. 0.5.0 routed PDF through the third-party `liteparse` engine, and js-chunks carried an `@llamaindex/liteparse-wasm` peer dependency to do the same in the browser. Both are gone. `liteparse` survives in `rs-chunks` as an **optional** dependency behind the `pdf-native` Cargo feature, used for one thing only: rasterising a page with PDFium when a scanned PDF has neither extractable text nor an embedded page image. The feature is on by default for native builds and compiled out of the WASM build, which is the one deliberate capability difference between js-chunks and the other two SDKs. Every SDK, including js-chunks, parses PDF text with the engine's own parser. - Standard-14 font metrics shipped: fonts that declare no `/Widths` no longer advance every glyph half an em, which had interleaved adjacent text runs (`Science, University of Oxford` → `ScieUnnciev,ersity of Oxford`). - Subset fonts naming glyphs by index (`/g18`) now decode — one 24-page paper went from 7,921 to 46,990 extracted characters. - Genuine two-column pages are split into columns and read down them; tables and bylines are told apart from columns and kept intact. - Author blocks read down the column instead of across, so each author keeps their own affiliation and email. - `default` streams incrementally: the first chunk of a 5,000-page document costs fewer than 10 pages instead of a full parse. - Ligatures are spelled out (`classification` → `classification`) so words stay searchable, and hyperlink targets become Markdown links. **Legacy `.doc` / `.ppt`** — two offset bugs were corrupting the start and end of virtually every real `.doc`, and a third meant **no `.doc` ever had paragraph properties**, which silently disabled table, list, page-break and style detection. All are fixed, and `.doc` now reports tables, list depth and section breadcrumbs. Footnotes, headers, annotations, endnotes and text boxes are extracted; page and slide provenance reaches every mode. **Encoding and text** - `.txt` encoding is detected rather than assumed: UTF-16 via BOM and a NUL sniff, cp1252 as the 8-bit fallback, every BOM stripped. - Line endings are normalised once at decode — CRLF documents used to come back as one unsplit chunk, which affected every Windows-authored `.txt` and `.md`. - Literal `&` is no longer dropped from DOCX and PPTX text (`R&D` → `R D`), and every named entity is decoded across HTML, XML and OpenDocument. **Spreadsheets** — an unreadable sheet no longer aborts the workbook, workbooks that `calamine` rejects are repaired before reading, XLSX streams for real, and skipped sheets are named in the metadata. **Email, eBooks, notebooks, OpenDocument** - `.eml`/`.mbox`: text attachments inlined, threading headers surfaced, per-message identity on every mbox chunk, and the declared charset verified rather than trusted. - `.msg`: image attachments extracted, Exchange legacyDNs no longer formatted as email addresses, and non-Latin RTF bodies no longer lost. - `.epub`: repeated metadata kept, the native TOC parsed, navigation pages flagged. - `.ipynb`: images emit a placeholder in every mode, the richest output MIME wins. - `.odt`/`.odp`: numbered lists stay numbered, slides carry the identity `.pptx` chunks already had, images are referenced and deduped. **Metadata** — JSON chunks name the records they were built from (`record_range`), and the metadata reference documents every key per format and mode. ## 0.5.0 *Released 2026-07-28.* - **Seven new formats**, a **PDF liteparse rewrite** (PDF ran through the third-party `liteparse` engine, which vendors its own PDFium — replaced in 0.6.0 by the engine's own parser), and **image extraction for spreadsheets and HTML**. ## 0.4.7 - Image extraction for **DOC** and **PPT** (legacy binary formats) via MS-ODRAW BLIP records (JPEG/PNG). ## 0.4.6 - Documentation fixes: PPT in the package description; `.pdf` added to the `list_images` format list. ## 0.4.4 - **PPT (PowerPoint 97–2003)** support — a pure-Rust CFB parser, all 7 document modes, streaming, and `to_markdown`. ## 0.4.3 - **PDF image extraction** via `list_images=True`, with a scanned-PDF fallback. ## 0.4.2 - Image extraction extended to **XLSX** and **HTML**; `get_chunks(list_images=True)` and the `ChunksResult` API introduced. ## 0.4.1 - Image extraction via `get_markdown(list_images=True)` for **DOCX** and **PPTX**. ## 0.4.0 - **DOC (Word 97–2003)** support and `get_markdown()` for all formats. ## 0.3.1 - **CSV** support (`row`, `sliding_window`, `page_aware`). ## 0.3.0 - **XLSX / XLS** support with 6 spreadsheet modes. ## 0.2.0 - **Streaming** for all formats and a Rust module refactor.