chunk-engine

Quick Start

Chunk any document, stream large files, and convert to Markdown — in Python, JavaScript, or Rust.

View raw

Every code block on this page is language-aware — use the tabs (or the global switcher in the nav) to see Python, JavaScript, or Rust.

Batch chunking

Pass a source and, optionally, a mode. Each chunk has content, a typed content_type, and metadata.

from py_chunks import get_chunks, stream_chunks, get_markdown

# Batch — works for every supported format
chunks = get_chunks("document.pdf")
chunks = get_chunks("notes.md",  mode="semantic")
chunks = get_chunks("deck.pptx", mode="sliding_window", window_size=3, overlap=1)

for chunk in chunks:
    print(chunk["content"], chunk["content_type"], chunk["metadata"])

# Streaming — constant memory over huge files
for chunk in stream_chunks("large.pdf", mode="section"):
    handle(chunk)

# Markdown conversion
md = get_markdown("report.docx")

Streaming

Yield chunks one at a time with constant memory — ideal for large files or streaming HTTP responses. Which modes stream and their per-format memory profile are in Streaming.

from py_chunks import stream_chunks

# Constant memory — yields one chunk at a time
for chunk in stream_chunks("data.csv", mode="row"):
    handle(chunk)

Markdown conversion

Convert any supported document to a Markdown string.

from py_chunks import get_markdown

md = get_markdown("report.docx")           # -> str
md = get_markdown(file_bytes, filename="report.pdf")  # bytes also supported

Image extraction

Ask for embedded images alongside the chunks.

from py_chunks import get_chunks, ChunksResult

result = get_chunks("deck.pptx", list_images=True)   # -> ChunksResult
result.chunks   # text chunks + image chunks (content_type="image")
result.images   # {"<hash>.jpeg": b"..."}

Image extraction is supported for DOCX, PPTX, XLSX, HTML, PDF, DOC, PPT, EPUB, IPYNB, EML, and ODT/ODP — see Supported Formats.

Next steps

On this page