# Python — py-chunks

Every name py-chunks exports, with exact signatures, return types and the exceptions each raises.

```bash
pip install py-chunks
```

The package is `py-chunks` on PyPI and imports as `py_chunks`. It ships
`py.typed` and a complete stub for the compiled module, so mypy and pyright see
full annotations. `py_chunks.__version__` reads the installed distribution's
metadata, so it cannot drift from what pip installed.

## What the package exports

`py_chunks.__all__` contains **55 names**, in five groups:

| Group | Count | Names |
| --- | --- | --- |
| Package metadata | 1 | `__version__` |
| Result types | 2 | `ChunksResult`, `MarkdownResult` |
| Source-agnostic entry points | 13 | `get_chunks`, `get_markdown`, `stream_chunks`, and the `_from_path` / `_from_bytes` / `_from_fileobj` / `_from_upload` / `_from_s3_presigned_url` families |
| Per-format chunkers | 36 | 18 `chunk_*` / `stream_chunk_*` pairs |

## Source-agnostic entry points

These are the 13 functions most callers use. All keyword arguments after `*`
are keyword-only.

### get_chunks

```python
def get_chunks(
    source,
    *,
    filename: str | None = None,
    mode: str = "default",
    window_size: int = 3,
    overlap: int = 1,
    sentences_per_chunk: int = 3,
    paragraphs_per_page: int = 15,
    delimiter: str | None = None,
    encoding: str = "auto",
    list_images: bool = False,
) -> list[dict] | ChunksResult
```

`source` may be a path (`str` or `os.PathLike`), an `http(s)` URL string,
`bytes` / `bytearray` / `memoryview`, a file-like object with `.read()`, or a
framework upload object with a `.filename` attribute. Returns `list[dict]`, or
a `ChunksResult` when `list_images=True` (typed via `@overload`, so a checker
narrows the return for a literal `True`/`False`).

### get_markdown

```python
def get_markdown(
    source,
    *,
    filename: str | None = None,
    list_images: bool = False,
) -> str | MarkdownResult
```

`get_markdown` takes no chunking parameters — it produces the document's
Markdown, not chunks. Returns `str`, or a `MarkdownResult` when
`list_images=True`.

### stream_chunks

```python
def stream_chunks(
    source,
    *,
    filename: str | None = None,
    mode: str = "default",
    window_size: int = 3,
    overlap: int = 1,
    sentences_per_chunk: int = 3,
    paragraphs_per_page: int = 15,
    delimiter: str | None = None,
    encoding: str = "auto",
)  # -> an iterator yielding chunk dicts
```

No `list_images` — streaming yields chunks only. See
[Streaming](/docs/streaming) for which formats are genuinely incremental.

### The explicit source helpers

Ten more functions skip source detection. Each mirrors the corresponding
`get_chunks` / `stream_chunks` signature above, differing only in the leading
positional parameters:

| Function | Leading parameters | Returns |
| --- | --- | --- |
| `get_chunks_from_path` | `file_path: str` | `list[dict] \| ChunksResult` |
| `get_chunks_from_bytes` | `data: bytes, filename: str` | `list[dict] \| ChunksResult` |
| `get_chunks_from_fileobj` | `file_obj, filename: str \| None = None` | `list[dict] \| ChunksResult` |
| `get_chunks_from_upload` | `upload_file` | `list[dict] \| ChunksResult` |
| `get_chunks_from_s3_presigned_url` | `url: str, filename: str \| None = None, timeout: int = 60` | `list[dict] \| ChunksResult` |
| `stream_chunks_from_path` | `file_path: str` | `Iterator[dict]` |
| `stream_chunks_from_bytes` | `data: bytes, filename: str` | `Iterator[dict]` |
| `stream_chunks_from_fileobj` | `file_obj, filename: str \| None = None` | `Iterator[dict]` |
| `stream_chunks_from_upload` | `upload_file` | `Iterator[dict]` |
| `stream_chunks_from_s3_presigned_url` | `url: str, filename: str \| None = None, timeout: int = 60` | `Iterator[dict]` |

Every one of them then takes the same keyword-only block:
`mode="default"`, `window_size=3`, `overlap=1`, `sentences_per_chunk=3`,
`paragraphs_per_page=15`, `delimiter=None`, `encoding="auto"` — plus
`list_images=False` on the `get_chunks_*` five.

`get_chunks_from_path` takes `file_path` positionally and does not accept a
`filename` override; the extension comes from the path.

  Since 0.6.1, `get_chunks_from_bytes` and the file-object / upload / URL paths
  that funnel into it call the engine's no-filesystem API directly. The one
  exception is `stream_chunks_from_bytes`, which still writes a temporary file
  because the engine's streaming surface is path-based; the temp file is
  deleted when the iterator is exhausted, closed, or garbage-collected.

## Per-format chunkers

18 modules, each exporting a `chunk_*` / `stream_chunk_*` pair. They take a
path (never bytes), skip source detection, and return a `(chunks, timing)`
tuple where `timing` is `{"rust_ms": float, "python_ms": float}`. The streaming
half returns an iterator of `dict`.

| Pair | Extensions | Signature after `file_path` |
| --- | --- | --- |
| `chunk_csv` / `stream_chunk_csv` | `.csv` | `mode="row", rows_per_chunk=10, window_size=5, overlap=1, include_headers=True, delimiter=None, encoding="auto", skip_empty_rows=True` |
| `chunk_tsv` / `stream_chunk_tsv` | `.tsv` | same as CSV, but `delimiter="\t"` |
| `chunk_doc` / `stream_chunk_doc` | `.doc` | standard |
| `chunk_docx` / `stream_chunk_docx` | `.docx`, `.docm`, `.dotx`, `.dotm` | standard |
| `chunk_eml` / `stream_chunk_eml` | `.eml`, `.mbox` | standard |
| `chunk_epub` / `stream_chunk_epub` | `.epub` | standard |
| `chunk_html` / `stream_chunk_html` | `.html`, `.htm` | standard |
| `chunk_ipynb` / `stream_chunk_ipynb` | `.ipynb` | standard |
| `chunk_json` / `stream_chunk_json` | `.json`, `.jsonl`, `.ndjson` | standard |
| `chunk_md` / `stream_chunk_md` | `.md` | standard |
| `chunk_msg` / `stream_chunk_msg` | `.msg` | standard |
| `chunk_odf` / `stream_chunk_odf` | `.odt`, `.odp` | standard |
| `chunk_pdf` / `stream_chunk_pdf` | `.pdf` | standard |
| `chunk_ppt` / `stream_chunk_ppt` | `.ppt` | standard |
| `chunk_pptx` / `stream_chunk_pptx` | `.pptx`, `.potx`, `.potm`, `.ppsx`, `.ppsm` | standard, but `paragraphs_per_page=5` and an extra `slides_per_chunk: int \| None = None` |
| `chunk_rtf` / `stream_chunk_rtf` | `.rtf` | standard |
| `chunk_txt` / `stream_chunk_txt` | `.txt` | standard |
| `chunk_xlsx` / `stream_chunk_xlsx` | `.xlsx`, `.xls`, `.xlsm`, `.xlsb`, `.ods`, `.xltx`, `.xltm` | see below |

**standard** means
`mode="default", window_size=3, overlap=1, sentences_per_chunk=3, paragraphs_per_page=15`.

There is no `chunk_xls`: `.xls` is one of the seven spreadsheet extensions
routed through `chunk_xlsx`.

### The delimited and spreadsheet signatures in full

```python
def chunk_csv(
    file_path: str,
    mode: str = "row",
    rows_per_chunk: int = 10,
    window_size: int = 5,
    overlap: int = 1,
    include_headers: bool = True,
    delimiter: str | None = None,
    encoding: str = "auto",
    skip_empty_rows: bool = True,
) -> tuple[list[dict], dict]

def stream_chunk_csv(...)   # identical signature, returns an iterator
```

Modes accepted: `row`, `default`, `sliding_window`, `page_aware`. `delimiter`
accepts only `None`, `","`, `"\t"`, `";"` or `"|"`.

```python
def chunk_xlsx(
    file_path: str,
    mode: str = "row",
    rows_per_chunk: int = 1,
    window_size: int = 3,
    overlap: int = 1,
    include_headers: bool = True,
    sheet_names: list[str] | None = None,
    skip_empty_rows: bool = True,
    serialize_as: str = "key_value",
    max_chunk_chars: int = 2000,
) -> tuple[list[dict], dict]

def stream_chunk_xlsx(
    file_path: str,
    mode: str = "row",
    rows_per_chunk: int = 1,
    window_size: int = 3,
    overlap: int = 1,
    include_headers: bool = True,
    sheet_names: list[str] | None = None,
    skip_empty_rows: bool = True,
    max_chunk_chars: int = 2000,
)
```

Modes accepted: `row`, `table`, `sheet`, `sliding_window`, `page_aware`,
`semantic`. `serialize_as` currently accepts only `"key_value"`, and it exists
on `chunk_xlsx` alone — the streaming half hard-codes it.

  `window_size` defaults to **5** in `chunk_csv` / `stream_chunk_csv` (and in
  the TSV wrappers, which inherit it) and to **3** in `chunk_xlsx` /
  `stream_chunk_xlsx` and in all sixteen other pairs. The source-agnostic
  `get_chunks` always uses 3. If you switch from `get_chunks` to `chunk_csv`
  without passing `window_size`, `sliding_window` output changes.

## Result types

Both are `dataclasses`, not tuples — access by attribute.

```python
@dataclass
class ChunksResult:
    chunks: list[dict]
    images: dict[str, bytes] = field(default_factory=dict)

@dataclass
class MarkdownResult:
    markdown: str
    images: dict[str, bytes] = field(default_factory=dict)
```

`images` is a `dict` keyed by image name — the JavaScript SDK returns the same
data as an **array** of `{ name, data }`. See
[Output Schema](/docs/output-schema).

## Exceptions

py-chunks defines no exception classes of its own; every failure is a Python
built-in. The table below covers the ones you will see in practice, but it is
not a closed set — treat any `Exception` subclass as possible.

| Exception | Raised when |
| --- | --- |
| `FileNotFoundError` | The path does not exist — checked in the Python layer before the engine is called. |
| `ValueError` | Unsupported extension; unknown `mode`; an invalid parameter (`overlap >= window_size`, a zero count, an unsupported `delimiter`); empty `data`; a byte or file-object source with no resolvable `filename`. |
| `TypeError` | `source` is a type the package does not accept; `.read()` returned something that is not `bytes`; an upload object whose `read()` is a coroutine. |
| `NotImplementedError` | `Streaming not yet supported for {ext} files` — the extension has no streaming entry point. All 36 supported extensions have one, so this is currently unreachable. **A mis-typed `mode` no longer lands here**: the Word family used to report `Streaming for <mode> mode coming soon`, which was doubly misleading (all seven DOCX modes stream fine), and it now raises `ValueError` like every other format. |
| `RuntimeError` | The engine failed to parse or decode the document. |
| `OSError` | An I/O failure inside the engine. |
| `OverflowError` | A **negative** count reached the Rust boundary before any Python-layer range check could reject it — `can't convert negative int to unsigned`. Seen on the formats whose validation lives in the engine: `.json`, `.jsonl`, `.ndjson`, `.eml`, `.mbox`, `.msg`, `.rtf`, `.ipynb`, `.odt`, `.odp`, `.epub`. The same negative value is a plain `ValueError` on `.md`, `.docx`, `.pdf` and friends. |

`RuntimeError`, `OSError` and `OverflowError` come from the Rust boundary. The
engine's error type has four variants, mapped as:

| Engine variant | Python exception |
| --- | --- |
| `ChunkError::Unsupported` | `ValueError` |
| `ChunkError::InvalidArg` | `ValueError` |
| `ChunkError::Parse` | `RuntimeError` |
| `ChunkError::Io` | `OSError` |
| any future variant | `RuntimeError` |

For **parse** errors the engine raises, the message text is the engine's own,
unprefixed — which is what makes it byte-identical to the message js-chunks
reports for the same input.

Other checks run earlier, in py-chunks' own Python layer. Two of those messages
are py-chunks-specific: the unsupported-extension message (which appends the
full supported set) and the invalid-mode message (one uniform sorted-list
template, where js-chunks reports the engine's per-format sentence).

The **numeric-range** messages are not among them any more. The formats whose
range checks live in Python (DOCX, DOC, PPT, PPTX and the spreadsheet family)
now use the engine's own wording verbatim, so py-chunks and js-chunks both say
`window_size must be greater than 0` — and so do the document and spreadsheet
families as each other. Only where a format renames the parameter does the
sentence change, and it changes identically in both SDKs (CSV/TSV report
`rows_per_chunk must be greater than 0` for a zero `paragraphs_per_page`).

The exception *type* matches js-chunks' `kind` in every case, so **branch on
the exception type, not on the message** — that stays the right habit even now
that the range wording agrees, because the two host-layer messages above still
do not. See [Error Handling](/docs/error-handling).

## `fit_tokens` — fit chunks to a token budget

```python
fit_tokens(
    chunks, counter, budget, *,
    overlap=0,                    # int tokens, or a fraction of budget (Python: 0 <= f < 1)
    tokenizer_kwargs=None,        # forwarded to the tokenizer's encode()
    min_tokens=0,
    merge="forward",              # "forward" | "none"
    split="sentence",             # "sentence" | "paragraph" | "hard"
    merge_metadata="first",       # "first" | "union"
    oversize="split",             # "split" | "keep" | "error"
    respect_boundaries=True,
    boundary_keys=("section_heading", "page_number", "sheet_name"),
) -> list[dict]
```

Takes `get_chunks` output and re-fits it so **no chunk exceeds `budget`** under
*your* tokenizer. That guarantee holds for every input, including
whitespace-free text — CJK prose, long URLs, base64 — because the split ladder
bottoms out at a character-level bisection sized by your own counter. (Before
0.6.4 an indivisible whitespace-free run was silently emitted over budget.)

`counter` is anything you're likely to be holding — coercion is built in:

```python

enc = tiktoken.get_encoding("cl100k_base")
chunks = py_chunks.get_chunks("report.pdf")

fitted = py_chunks.fit_tokens(chunks, enc, 512)            # an Encoding
fitted = py_chunks.fit_tokens(chunks, "cl100k_base", 512)  # an encoding name
fitted = py_chunks.fit_tokens(chunks, "gpt-4", 512)        # a model name
fitted = py_chunks.fit_tokens(chunks, hf_tokenizer, 512)   # a HF tokenizer
fitted = py_chunks.fit_tokens(chunks, my_callable, 512)    # str -> int
```

Name strings resolve through tiktoken first, then `transformers` (each imported
lazily, only if needed); HuggingFace counters default to
`add_special_tokens=False` so BOS/EOS tokens are not billed to every chunk —
override via `tokenizer_kwargs`. `overlap` shares tokens between the parts of a
chunk that had to be split; it is deliberately scoped *within* one engine chunk
(overlapping across chunks would smear their metadata and structural
boundaries), and overlapped parts still respect `budget` — that is asserted,
not assumed.

The same coercion is exported as `py_chunks.coerce_counter(counter,
tokenizer_kwargs=None)` if you want the `str -> int` for your own use.

## `chunk_text` — chunk a bare string

```python
chunk_text(text, *, mode="default", window_size=3, overlap=1,
           sentences_per_chunk=3, paragraphs_per_page=15) -> list[dict]
```

No file, no extension: the string runs through the plain-text pipeline exactly
as a `.txt` file's bytes would, so every mode and metadata key matches `.txt`
behaviour. (`get_chunks("some prose")` deliberately raises — a path-taking API
must never guess whether a string is a path or a document.) Composes with
`fit_tokens` for a token budget:

```python
fitted = py_chunks.fit_tokens(py_chunks.chunk_text(text), "cl100k_base", 512)
```

The chunks keep their `content_type` and `metadata`, including the repeated
table headers that make a retrieved row interpretable — which is the difference
between this and running a text splitter over the raw document.

  Every other API in this library produces byte-identical output across
  py-chunks, js-chunks and rs-chunks. **`fit_tokens` does not, and cannot.**
  Its output depends on the token counter you pass, and tokenizers differ
  between languages — so putting a token budget in the engine would either
  vendor the wrong tokenizer for your embedding model or quietly break the
  byte-identical guarantee. Characters are parity-safe; tokens are not.

  `js-chunks` ships the same helper as
  [`fitTokens`](/docs/api-reference/javascript#fittokens) — a separate,
  equally parity-exempt implementation, not a shared one.

### Parameters worth knowing

| parameter | why you'd change it |
|---|---|
| `min_tokens` | Merge chunks below this into the next one. `0` (default) disables merging entirely — only oversized chunks are touched. |
| `respect_boundaries` | On by default: never merge across a change of `section_heading`, `page_number` or `sheet_name`. Merging across them would make the surviving metadata a lie, and is the fastest way to destroy table-header repetition. |
| `merge_metadata` | `"first"` keeps the leading chunk's values and never invents one. `"union"` collects differing values into a list, so a chunk spanning two pages reports both rather than claiming one. |
| `oversize` | What to do when a single indivisible piece still exceeds the budget: `"split"` cuts it, `"keep"` emits it whole, `"error"` raises naming the chunk. |

An unrecognised value for `merge`, `split`, `merge_metadata` or `oversize`
raises `ValueError` rather than falling through to a default — a typo like
`split="sentances"` would otherwise quietly pick a different strategy.

Split chunks gain `fit_part` / `fit_total` in their metadata. The input list is
never mutated.

  It is tempting to raise `min_tokens` to pack chunks fuller. Measured on this
  project's own corpus, raising budget utilisation from 46% to 95% **halved
  token-level precision** with no gain in answerability. Use `min_tokens` to
  remove genuinely useless fragments, not to fill the budget.

## Related

- [Input Sources](/docs/input-sources) — every accepted `source` shape, with examples.
- [Chunking Modes](/docs/chunking-modes) — what each `mode` produces.
- [Output Schema](/docs/output-schema) — the `dict` a chunk actually is.