Streaming

stream_chunks / streamChunks (and per-format stream(...) in Rust) hand you chunks one at a time instead of one list at the end. That lets you forward, persist or embed each chunk as it arrives.

Three classes of streaming

The matrix

All 36 supported extensions, by runtime. DEFAULT parameters assumed.

That is 36 extensions: 1 PDF + 7 spreadsheet + 2 delimited + 4 Word OOXML + 22 in the last row.

¹ Rust's CSV stream hands chunks to the consumer over an unbounded channel (PDF's is bounded at 64). The file is never fully read into memory, but a consumer slower than the reader can let produced chunks accumulate. If that matters, do your per-chunk work synchronously inside the loop rather than buffering.

The PDF exception, explained

PDF's default mode is the one place a document format truly streams, and the reason is heading ranking, not pagination:

• default ranks type sizes within each page, so a page can be parsed, chunked and dropped. With a resumable builder a chunk costs only the pages it came from — the first chunk of a 5,000-page PDF reads fewer than 10 of them.
• The other six modes rank sizes across the whole document. Their reader has read every page before it renders the first, and no amount of chunker work changes that.

For those six modes the engine still does the honest thing available to it: the parse runs on a worker thread and chunks arrive through a bounded channel of depth 64, so construction returns immediately and at most 64 chunks sit between producer and consumer. It is not bounded parse memory, but it is not a materialized list either.

Pagination is not the obstacle it looks like. The end of page 3 and the start of page 4 are routinely one paragraph, and chunking each page separately breaks the sentence at every boundary — measured on a 12-page paper, 71 chunks instead of 66. default does not chunk pages separately: the chunker is fed one continuous stream of markdown and finalizes a chunk only once the text after it has arrived.

What JavaScript's streamChunks actually is

export async function* streamChunks(source, opts = {}) {
  const chunks = await getChunks(source, { ...opts, listImages: false });
  for (const chunk of chunks) yield chunk;
}

That is the whole implementation. The WASM boundary is a synchronous full parse — there is no way to yield from inside it — so true streaming in JavaScript would be an engine redesign, not a wrapper change. This is a deliberate, documented decision, not an oversight.

Use it for the ergonomics: for await reads better than indexing an array, you can break out of the loop, and your await upsert(...) overlaps with nothing in particular but keeps the code shape identical to the Python and Rust versions. Do not use it to survive a file that would OOM getChunks.

from py_chunks import stream_chunks

# Genuinely incremental for pdf and xlsx: chunks are produced as the document
# is read, so breaking early skips the rest of the parse. Every other format
# is chunked in full by the binding and drained through the same iterator —
# same results, same peak memory as get_chunks().
batch = []
for chunk in stream_chunks("large.pdf", mode="section"):
    batch.append(chunk)
    if len(batch) == 128:
        upsert(batch)
        batch.clear()

if batch:
    upsert(batch)

Byte-identical to batch

Per-language shape

from py_chunks import stream_chunks

# Yields one chunk at a time
for chunk in stream_chunks("data.csv", mode="row"):
    handle(chunk)

Python's ChunkStreamIterator converts each chunk into a Python dict when you ask for it, not up front. On a 5,000-page PDF that is the difference between materializing 71,111 dicts before you read the first and building them one at a time — a real saving even for the eager-drain formats, where the Rust side has already finished.

Python also exposes source-specific streaming helpers — stream_chunks_from_path, stream_chunks_from_bytes, stream_chunks_from_fileobj, stream_chunks_from_upload, and stream_chunks_from_s3_presigned_url — mirroring the batch helpers in Input Sources.

Every one of the 36 supported extensions has a streaming entry point, so NotImplementedError: Streaming not yet supported for {ext} files is only reachable if a future format lands without one.

Choosing between streaming and batch

Bytes sources & cleanup

When you stream from bytes (e.g. a request body), Python writes a temporary file with the right extension — the engine's streaming paths are path-based — and removes it when the iterator is exhausted or you exit early. Use it as a context manager to guarantee cleanup:

with stream_chunks(data, filename="big.pdf", mode="section") as it:
    for chunk in it:
        ...

See Framework Integration for FastAPI, Express, Next.js and Axum handlers that forward NDJSON as chunks are produced.