Recipes
Practical, copy-pasteable patterns for RAG ingestion, batch processing, and streaming — in Python, JavaScript, or Rust.
Common end-to-end patterns. Every snippet is language-aware — use the tabs (or the global switcher in the nav) to see Python, JavaScript, or Rust.
RAG ingestion
Chunk a document into semantically coherent passages, embed each one, and store
it with its metadata. semantic mode gives the best embedding boundaries.
from py_chunks import get_chunks
def ingest(path):
# Semantic chunks make the best embeddings.
for chunk in get_chunks(path, mode="semantic"):
vector = embed(chunk["content"]) # your embedding model
store(vector, text=chunk["content"], meta=chunk["metadata"])Each chunk carries typed content_type and metadata (section headings, page
numbers, …), so you can filter results or show provenance at query time — see
the Output Schema.
Batch a folder of mixed formats
The same call handles every format, so a mixed folder just works. Isolate failures per file so one bad document doesn't halt the batch.
from pathlib import Path
from py_chunks import get_chunks
for path in Path("docs").rglob("*"):
if not path.is_file():
continue
try:
index(path.name, get_chunks(str(path)))
except Exception as e:
print("skip", path, e) # one bad file shouldn't halt the batchStream a large file
stream_chunks / streamChunks yield chunks one at a time with constant
memory — ideal for large files or streaming an HTTP response.
from py_chunks import stream_chunks
# Constant memory — yields one chunk at a time
for chunk in stream_chunks("data.csv", mode="row"):
handle(chunk)Streaming yields exactly the same chunks as the batch call, just incrementally.
In Rust, streaming is per-format via chunks_rs::formats::*::stream(...).
Convert to Markdown
Get a Markdown string from any supported document (for previews, diffing, or feeding a Markdown-native tool).
from py_chunks import get_markdown
md = get_markdown("report.docx") # -> str
md = get_markdown(file_bytes, filename="report.pdf") # bytes also supportedSee also Framework Integration for FastAPI, Flask, Django, Express, and Axum handlers.