Input Sources

All three SDKs dispatch by file extension. A path carries its own extension; every byte source needs a filename so the engine knows what it is looking at. Nothing else about the source changes the output — the same bytes produce the same chunks however they arrive.

What each SDK accepts

One table, all three SDKs. "Batch" is the whole-result call; "Stream" is the one-chunk-at-a-time iterator.

Python has the widest surface because it is the SDK that sits behind web frameworks; JavaScript and Rust expect you to produce the bytes and hand them over. Neither of those is a capability gap in the engine — every source below ends in the same two engine calls.

Python's source-agnostic surface is 13 functions: get_chunks and five get_chunks_from_*, stream_chunks and five stream_chunks_from_*, plus get_markdown. The get_chunks / stream_chunks pair sniffs the source and forwards to the specific one, so you rarely need the others by name — but they exist, they are exported, and calling them directly skips the sniffing.

from py_chunks import (
    get_chunks,                            # sniffs any of the five below
    get_chunks_from_path,
    get_chunks_from_bytes,
    get_chunks_from_fileobj,
    get_chunks_from_upload,
    get_chunks_from_s3_presigned_url,
    stream_chunks,                         # sniffs any of the five below
    stream_chunks_from_path,
    stream_chunks_from_bytes,
    stream_chunks_from_fileobj,
    stream_chunks_from_upload,
    stream_chunks_from_s3_presigned_url,
    get_markdown,
)

get_chunks picks a branch in this order: str/os.PathLike (an http/https scheme routes to the URL helper, anything else to the path helper) → memoryview/bytearray/bytes → an object with a .filename attribute (upload) → an object with .read() (file-like) → TypeError. Note that the upload check comes before the file-like check, so any object carrying a filename attribute takes the upload path.

Filesystem path

from py_chunks import get_chunks, get_chunks_from_path

# get_chunks() sniffs the source type; get_chunks_from_path() is the explicit one.
chunks = get_chunks("notes.md")
chunks = get_chunks_from_path("notes.md", mode="section")

# A missing path raises FileNotFoundError before any parsing happens.

get_chunks_from_path checks the file exists first and raises FileNotFoundError(f"File not found: {file_path}") before any parsing happens. str and pathlib.Path both work (anything implementing os.PathLike does).

Only http and https are recognised as URL schemes — an s3:// or file:// string falls through to the path branch and fails as a missing file.

A bare string

New in 0.6.4: text you already hold as a string chunks directly, with no file, no extension, and no encoding dance.

from py_chunks import chunk_text

chunks = chunk_text("raw text you already have in memory...")

import { chunkText } from "js-chunks";

const chunks = await chunkText(rawText);

The string is routed through the plain-text pipeline, so it returns the same structure-aware chunks a .txt file would, takes the same mode parameters, and composes with fit_tokens / fitTokens.

Strings stay unambiguous everywhere else: get_chunks("...") still treats a string as a path and chunk_text treats it as content — one meaning per function, so a filename-looking document can never be mistaken for a file to open. In Rust, encode and use get_chunks_from_bytes(text.as_bytes(), "text.txt", …).

Raw bytes

The common case: you already hold the document in memory and never want it on disk.

from py_chunks import get_chunks_from_bytes

with open("report.pdf", "rb") as f:
    data = f.read()

# Straight to the engine's no-filesystem API — nothing is written to disk.
# filename is used only for extension detection.
chunks = get_chunks_from_bytes(data, "report.pdf")

get_chunks(data, filename=...) and get_chunks_from_bytes(data, filename) are the same call; the latter makes filename positional and required. bytearray and memoryview are converted for you.

Two errors to expect:

• ValueError("filename is required when source is bytes") — from get_chunks when you omit it.
• ValueError("data is empty") — zero-length input is rejected up front rather than parsed into nothing.

Blob and File

File-like objects

get_chunks_from_fileobj(file_obj, filename=None) accepts anything with .read() — an open file, a BytesIO, a tempfile, a socket-backed reader.

import io
from py_chunks import get_chunks_from_fileobj

# An open file carries its own .name — no filename argument needed.
with open("notes.md", "rb") as f:
    chunks = get_chunks_from_fileobj(f)

# A BytesIO does not, so name it. (.read() may return str or bytes; both work.)
buf = io.BytesIO(b"# Chunking Notes\n\nHello.")
chunks = get_chunks_from_fileobj(buf, filename="notes.md")

Behaviour worth knowing:

• Filename inference is filename or getattr(file_obj, "name", None). An open() handle supplies it; a BytesIO does not, so name it or you get ValueError("filename is required when file object has no name").
• Text mode works: a str from .read() is re-encoded as UTF-8. That ignores the file's own encoding, so open binary files in "rb".
• The object is not rewound. .read() is called once, with no seek(0), so a stream you already consumed yields b"" and then ValueError("data is empty").

Framework uploads

get_chunks_from_upload(upload_file) takes a FastAPI / Starlette UploadFile, a Django UploadedFile, or anything else with a .filename attribute.

from fastapi import FastAPI, UploadFile
from py_chunks import get_chunks_from_upload

app = FastAPI()

@app.post("/chunk")
def chunk(file: UploadFile):
    # Reads upload_file.file (the SpooledTemporaryFile) when present, because
    # UploadFile.read() is a coroutine — passing the object itself is safe in a
    # sync handler. filename comes from upload_file.filename.
    return get_chunks_from_upload(file, mode="section")

The filename comes from upload_file.filename; a missing or empty one raises ValueError("upload_file.filename is required"). See Framework Integration for complete handlers.

URLs and pre-signed links

from py_chunks import get_chunks, get_chunks_from_s3_presigned_url

# Downloads with urlopen, then chunks the bytes in memory (nothing hits disk).
chunks = get_chunks_from_s3_presigned_url(url, timeout=60)

# The filename defaults to the last URL path segment; override it when the
# URL has no useful name (pre-signed links often don't).
chunks = get_chunks_from_s3_presigned_url(url, filename="report.pdf")

# get_chunks() routes http/https sources here for you.
chunks = get_chunks(url, filename="report.pdf")

get_chunks_from_s3_presigned_url(url, filename=None, timeout=60) downloads with urllib.request.urlopen and chunks the bytes in memory. get_chunks routes any http/https string here automatically.

The filename defaults to the last path segment of the URL, with the query string stripped — so an S3 signature does not end up looking like an extension. If the path has no final segment you get ValueError("filename is required when URL path has no filename"); pass filename= for pre-signed links that carry an opaque key.

Streaming from a non-path source

from py_chunks import stream_chunks_from_bytes

# The engine's streaming surface is path-based, so this one source *does* touch
# disk: the bytes go to a NamedTemporaryFile that is deleted when the iterator
# is exhausted, closed, or the with-block exits. (Batch bytes never touch disk.)
with stream_chunks_from_bytes(data, "report.pdf", mode="section") as chunks:
    for chunk in chunks:
        handle(chunk)

list_images is a batch-only option — no stream_* function accepts it.

Markdown you already parsed

Bundled browsers

Next steps

• Output Schema — what comes back, whatever went in
• Streaming — the honest per-format, per-runtime matrix
• Error Handling — what each bad source raises
• Framework Integration — FastAPI, Express, Axum
• API Reference — every signature, per language