Python — py-chunks

pip install py-chunks

The package is py-chunks on PyPI and imports as py_chunks. It ships py.typed and a complete stub for the compiled module, so mypy and pyright see full annotations. py_chunks.__version__ reads the installed distribution's metadata, so it cannot drift from what pip installed.

What the package exports

py_chunks.__all__ contains 55 names, in five groups:

Source-agnostic entry points

These are the 13 functions most callers use. All keyword arguments after * are keyword-only.

get_chunks

def get_chunks(
    source,
    *,
    filename: str | None = None,
    mode: str = "default",
    window_size: int = 3,
    overlap: int = 1,
    sentences_per_chunk: int = 3,
    paragraphs_per_page: int = 15,
    delimiter: str | None = None,
    encoding: str = "auto",
    list_images: bool = False,
) -> list[dict] | ChunksResult

source may be a path (str or os.PathLike), an http(s) URL string, bytes / bytearray / memoryview, a file-like object with .read(), or a framework upload object with a .filename attribute. Returns list[dict], or a ChunksResult when list_images=True (typed via @overload, so a checker narrows the return for a literal True/False).

get_markdown

def get_markdown(
    source,
    *,
    filename: str | None = None,
    list_images: bool = False,
) -> str | MarkdownResult

get_markdown takes no chunking parameters — it produces the document's Markdown, not chunks. Returns str, or a MarkdownResult when list_images=True.

stream_chunks

def stream_chunks(
    source,
    *,
    filename: str | None = None,
    mode: str = "default",
    window_size: int = 3,
    overlap: int = 1,
    sentences_per_chunk: int = 3,
    paragraphs_per_page: int = 15,
    delimiter: str | None = None,
    encoding: str = "auto",
)  # -> an iterator yielding chunk dicts

No list_images — streaming yields chunks only. See Streaming for which formats are genuinely incremental.

The explicit source helpers

Ten more functions skip source detection. Each mirrors the corresponding get_chunks / stream_chunks signature above, differing only in the leading positional parameters:

Every one of them then takes the same keyword-only block: mode="default", window_size=3, overlap=1, sentences_per_chunk=3, paragraphs_per_page=15, delimiter=None, encoding="auto" — plus list_images=False on the get_chunks_* five.

get_chunks_from_path takes file_path positionally and does not accept a filename override; the extension comes from the path.

Per-format chunkers

18 modules, each exporting a chunk_* / stream_chunk_* pair. They take a path (never bytes), skip source detection, and return a (chunks, timing) tuple where timing is {"rust_ms": float, "python_ms": float}. The streaming half returns an iterator of dict.

standard means mode="default", window_size=3, overlap=1, sentences_per_chunk=3, paragraphs_per_page=15.

There is no chunk_xls: .xls is one of the seven spreadsheet extensions routed through chunk_xlsx.

The delimited and spreadsheet signatures in full

def chunk_csv(
    file_path: str,
    mode: str = "row",
    rows_per_chunk: int = 10,
    window_size: int = 5,
    overlap: int = 1,
    include_headers: bool = True,
    delimiter: str | None = None,
    encoding: str = "auto",
    skip_empty_rows: bool = True,
) -> tuple[list[dict], dict]

def stream_chunk_csv(...)   # identical signature, returns an iterator

Modes accepted: row, default, sliding_window, page_aware. delimiter accepts only None, ",", "\t", ";" or "|".

def chunk_xlsx(
    file_path: str,
    mode: str = "row",
    rows_per_chunk: int = 1,
    window_size: int = 3,
    overlap: int = 1,
    include_headers: bool = True,
    sheet_names: list[str] | None = None,
    skip_empty_rows: bool = True,
    serialize_as: str = "key_value",
    max_chunk_chars: int = 2000,
) -> tuple[list[dict], dict]

def stream_chunk_xlsx(
    file_path: str,
    mode: str = "row",
    rows_per_chunk: int = 1,
    window_size: int = 3,
    overlap: int = 1,
    include_headers: bool = True,
    sheet_names: list[str] | None = None,
    skip_empty_rows: bool = True,
    max_chunk_chars: int = 2000,
)

Modes accepted: row, table, sheet, sliding_window, page_aware, semantic. serialize_as currently accepts only "key_value", and it exists on chunk_xlsx alone — the streaming half hard-codes it.

Result types

Both are dataclasses, not tuples — access by attribute.

@dataclass
class ChunksResult:
    chunks: list[dict]
    images: dict[str, bytes] = field(default_factory=dict)

@dataclass
class MarkdownResult:
    markdown: str
    images: dict[str, bytes] = field(default_factory=dict)

images is a dict keyed by image name — the JavaScript SDK returns the same data as an array of { name, data }. See Output Schema.

Exceptions

py-chunks defines no exception classes of its own; every failure is a Python built-in. The table below covers the ones you will see in practice, but it is not a closed set — treat any Exception subclass as possible.

RuntimeError, OSError and OverflowError come from the Rust boundary. The engine's error type has four variants, mapped as:

For parse errors the engine raises, the message text is the engine's own, unprefixed — which is what makes it byte-identical to the message js-chunks reports for the same input.

Other checks run earlier, in py-chunks' own Python layer. Two of those messages are py-chunks-specific: the unsupported-extension message (which appends the full supported set) and the invalid-mode message (one uniform sorted-list template, where js-chunks reports the engine's per-format sentence).

The numeric-range messages are not among them any more. The formats whose range checks live in Python (DOCX, DOC, PPT, PPTX and the spreadsheet family) now use the engine's own wording verbatim, so py-chunks and js-chunks both say window_size must be greater than 0 — and so do the document and spreadsheet families as each other. Only where a format renames the parameter does the sentence change, and it changes identically in both SDKs (CSV/TSV report rows_per_chunk must be greater than 0 for a zero paragraphs_per_page).

The exception type matches js-chunks' kind in every case, so branch on the exception type, not on the message — that stays the right habit even now that the range wording agrees, because the two host-layer messages above still do not. See Error Handling.

fit_tokens — fit chunks to a token budget

fit_tokens(
    chunks, counter, budget, *,
    overlap=0,                    # int tokens, or a fraction of budget (Python: 0 <= f < 1)
    tokenizer_kwargs=None,        # forwarded to the tokenizer's encode()
    min_tokens=0,
    merge="forward",              # "forward" | "none"
    split="sentence",             # "sentence" | "paragraph" | "hard"
    merge_metadata="first",       # "first" | "union"
    oversize="split",             # "split" | "keep" | "error"
    respect_boundaries=True,
    boundary_keys=("section_heading", "page_number", "sheet_name"),
) -> list[dict]

Takes get_chunks output and re-fits it so no chunk exceeds budget under your tokenizer. That guarantee holds for every input, including whitespace-free text — CJK prose, long URLs, base64 — because the split ladder bottoms out at a character-level bisection sized by your own counter. (Before 0.6.4 an indivisible whitespace-free run was silently emitted over budget.)

counter is anything you're likely to be holding — coercion is built in:

import py_chunks, tiktoken

enc = tiktoken.get_encoding("cl100k_base")
chunks = py_chunks.get_chunks("report.pdf")

fitted = py_chunks.fit_tokens(chunks, enc, 512)            # an Encoding
fitted = py_chunks.fit_tokens(chunks, "cl100k_base", 512)  # an encoding name
fitted = py_chunks.fit_tokens(chunks, "gpt-4", 512)        # a model name
fitted = py_chunks.fit_tokens(chunks, hf_tokenizer, 512)   # a HF tokenizer
fitted = py_chunks.fit_tokens(chunks, my_callable, 512)    # str -> int

Name strings resolve through tiktoken first, then transformers (each imported lazily, only if needed); HuggingFace counters default to add_special_tokens=False so BOS/EOS tokens are not billed to every chunk — override via tokenizer_kwargs. overlap shares tokens between the parts of a chunk that had to be split; it is deliberately scoped within one engine chunk (overlapping across chunks would smear their metadata and structural boundaries), and overlapped parts still respect budget — that is asserted, not assumed.

The same coercion is exported as py_chunks.coerce_counter(counter, tokenizer_kwargs=None) if you want the str -> int for your own use.

chunk_text — chunk a bare string

chunk_text(text, *, mode="default", window_size=3, overlap=1,
           sentences_per_chunk=3, paragraphs_per_page=15) -> list[dict]

No file, no extension: the string runs through the plain-text pipeline exactly as a .txt file's bytes would, so every mode and metadata key matches .txt behaviour. (get_chunks("some prose") deliberately raises — a path-taking API must never guess whether a string is a path or a document.) Composes with fit_tokens for a token budget:

fitted = py_chunks.fit_tokens(py_chunks.chunk_text(text), "cl100k_base", 512)

The chunks keep their content_type and metadata, including the repeated table headers that make a retrieved row interpretable — which is the difference between this and running a text splitter over the raw document.

Parameters worth knowing

An unrecognised value for merge, split, merge_metadata or oversize raises ValueError rather than falling through to a default — a typo like split="sentances" would otherwise quietly pick a different strategy.

Split chunks gain fit_part / fit_total in their metadata. The input list is never mutated.

Related

• Input Sources — every accepted source shape, with examples.
• Chunking Modes — what each mode produces.
• Output Schema — the dict a chunk actually is.