Chunk your first document, pick a mode, stream large files, and convert to Markdown — in Python, JavaScript, or Rust.

Prerequisite: the SDK for your language, installed. Everything on this page works with a default install and no configuration.

$pip install py-chunks

Showing examples for Python

— your choice follows you across the docs.

Your first chunks

Point the engine at a file. That is the whole API for the common case — there is nothing to configure and no pipeline to assemble.

first_chunks.py

from py_chunks import get_chunks

chunks = get_chunks("notes.md")
for c in chunks[:3]:
    print(c["content_type"], "|", c["content"][:40])

# heading | Chunking Notes
# plain_paragraph | Chunk-engine splits a document into retr
# heading | Why structure matters

Three fields come back on every chunk: content (the text), content_type (what kind of block it was), and metadata (headings, indices, counts). See Output Schema for the full shape.

Pick a mode

The mode argument decides how the document is cut up. It is the one knob worth setting deliberately.

I want to…	Mode	Typical use
Feed coherent passages to an embedding model	`semantic`	RAG over prose
Keep everything under a heading together	`section`	Document search
Index each paragraph, heading and table separately	`default` / `structural`	Fine-grained retrieval
Fixed sentence count per chunk	`sentence`	Uniform chunk sizes
Overlapping windows	`sliding_window`	Dense retrieval, recall-first
Preserve pages or slides for citations	`page_aware`	"see page 4" answers

Spreadsheets have their own modes (row, table, sheet); CSV and TSV take row, sliding_window and page_aware (and default, an alias for row), but not table or sheet. Start with semantic for LLM input and section for a search index; the full comparison, with real input→output examples, is in Chunking Modes.

semantic.py

from py_chunks import get_chunks

# Adjacent blocks merge when they look like the same idea. Every chunk records
# WHY it starts where it does, in primary_merge_reason (+ merge_reasons).
for c in get_chunks("notes.md", mode="semantic"):
    print(c["content_type"], "|", c["metadata"]["primary_merge_reason"])

# heading | initial
# semantic | initial
# heading | initial
# semantic | initial
# table | structural_boundary
# heading | initial
# semantic | initial

Streaming

Consume chunks one at a time instead of waiting for the whole list.

stream.py

from py_chunks import stream_chunks

# Yields one chunk at a time
for chunk in stream_chunks("data.csv", mode="row"):
    handle(chunk)

What streaming actually buys you

"Constant memory" is true for some formats and some runtimes — not all of them. Where it does not hold you still get lazy delivery, which lets you overlap embedding and upsert work with iteration; you do not get a lower memory ceiling.

Genuinely incremental for PDF and XLSX: chunks are produced as the document is read, and breaking early skips the rest of the parse. Every other format — CSV included — is chunked in full by the binding and drained through the same iterator, so peak memory matches get_chunks.

Streaming has the per-format, per-runtime matrix.

Markdown conversion

Convert any supported document to a Markdown string.

markdown.py

from py_chunks import get_markdown

md = get_markdown("report.docx")                       # -> str
md = get_markdown(file_bytes, filename="report.pdf")   # bytes also supported

# get_markdown does NOT accept URLs — a URL string is treated as a path and
# raises FileNotFoundError. Download it first, then pass the bytes.

Image extraction

Ask for embedded images alongside the chunks.

images.py

from py_chunks import get_chunks

result = get_chunks("deck.pptx", list_images=True)   # -> ChunksResult
result.chunks   # text chunks + image chunks (content_type="image")
result.images   # {"<hash>.jpeg": b"..."} — name matches the ![](name) reference

Image extraction is supported for DOC, DOCX (family), PPT, PPTX (family), XLSX (family, including ODS/XLSB — not XLS), HTML/HTM, PDF, EPUB, IPYNB, EML/MBOX, MSG, and ODT/ODP — see Supported Formats. Asking a format that has no image support returns an empty image collection rather than an error.

When it fails

Failures are typed, so you can tell "this file is not supported" from "this file is broken" without string matching.

errors.py

from py_chunks import get_chunks

try:
    chunks = get_chunks("report.xyz")
except FileNotFoundError as e:
    ...   # File not found: report.xyz
except ValueError as e:
    ...   # unsupported extension, bad mode, bad window/overlap
except RuntimeError as e:
    ...   # the document could not be parsed

# Real messages:
# ValueError: Unsupported file type '.xyz'. Supported: .csv, .doc, .docm, ...
# ValueError: mode must be one of ['default', 'page_aware', 'section',
#             'semantic', 'sentence', 'sliding_window', 'structural'] for MD,
#             got: 'nope'
# ValueError: overlap must be less than window_size
# ValueError: filename is required when source is bytes

Python maps engine failures onto built-in exceptions: FileNotFoundError for a missing path, ValueError for an unsupported extension / bad mode / bad window-overlap combination / missing filename on bytes, TypeError for a source type it cannot use, and RuntimeError when a document cannot be parsed.

Error Handling has the complete table.

Language notes

The source-agnostic surface is 13 functions. get_chunks and stream_chunks sniff the source type; the five *_from_path / *_from_bytes / *_from_fileobj / *_from_upload / *_from_s3_presigned_url variants of each are exported too if you would rather be explicit.
Uploads and URLs are first-class. get_chunks_from_upload(file) handles a FastAPI / Starlette / Django upload (read upload_file.file under the hood, so it is safe in a sync handler), and get_chunks("https://…") downloads and chunks in memory.
get_markdown is narrower than get_chunks: no URLs, no upload objects. See Input Sources.
Bytes never touch disk (since 0.6.1) — but streaming from bytes writes a temp file, because the engine's streaming surface is path-based.
Options are keyword-only after the source, and chunks are plain dicts: chunk["content_type"].

Next steps

Input Sources — paths, bytes, uploads, URLs, Blobs
Chunking Modes — pick the right strategy
Streaming — what streams incrementally, and what only looks like it
Framework Integration — FastAPI, Express, Axum, and more
API Reference — full signatures per language

Quick Start