chunk-engine

Supported Formats

All 36 file extensions the engine handles, the modes each supports, Markdown conversion, and image extraction.

View raw

py-chunks handles 36 file extensions through one API. Dispatch is by file extension, so the extension (or the filename you pass with bytes) must be correct.

All 36 ship in the same release

There is no tiering here. Every extension below is handled by the same shared engine and is present in py-chunks, js-chunks and rs-chunks from 0.6.1 onward — the same code path, the same output. If a format seems missing, you are on an older release: pip install -U py-chunks, npm i js-chunks@latest, cargo update.

By family

FamilyExtensions
Word.docx .doc .docm .dotx .dotm
PowerPoint.pptx .ppt .potx .potm .ppsx .ppsm
Spreadsheets.xlsx .xls .xlsm .xlsb .xltx .xltm .ods
PDF.pdf
Web & Markup.html .htm .md .rtf
OpenDocument.odt .odp (and .ods, above)
Plain & Data.txt .csv .tsv .json .jsonl .ndjson
Email.msg .eml .mbox
eBooks & Notebooks.epub .ipynb

For what each of these actually emits — content types, metadata keys, and real output blocks per family — see the Metadata Reference.

Chunking modes per format

  • Document formats (DOCX, DOC, PDF, PPTX, PPT, MD, HTML, TXT, MSG, EML/MBOX, ODT/ODP, JSON, RTF, EPUB, IPYNB): 7 modes — default, structural, section, semantic, sliding_window, sentence, page_aware.
  • Spreadsheets (XLSX, XLS, XLSM, XLSB, ODS, XLTX, XLTM): 6 modes — row, table, sheet, sliding_window, page_aware, semantic.
  • CSV / TSV: row (default), sliding_window, page_aware.

Markdown conversion

get_markdown supports documents, spreadsheets, and delimited files — Word, PowerPoint, PDF, HTML, Excel, CSV/TSV, TXT, Markdown, and the OpenDocument / email / eBook / notebook families. It does not accept URLs.

What each format produces

The conversion is format-aware — it maps each format's native structure to the closest Markdown construct rather than dumping plain text.

FormatMarkdown output
.docxFull fidelity: headings (####### from Word heading styles / outline levels), unordered (- item) and ordered (1. item) lists with per-level indentation, pipe tables, fenced code blocks, hyperlinks as [text](url), page/section breaks as ---, footnotes/endnotes as [^id]: text appended at the end.
.docH1 → #, H2–H3 → ##, H4+ → ### (levels come from the file's own paragraph styles); lists → - item, indented two spaces per nesting level; tables → one pipe table per table, rows padded to the widest and a single | --- | separator after the header row; page breaks → ---. Footnotes, headers, comments, endnotes and text boxes are appended under a # [Label] heading each.
.pptxPresentation title → # Title; PPTX sections → # Section; each slide → ## Slide N: Title; bullets (- item / 1. item) with indentation; pipe tables; speaker notes as > **Notes:** …; slides/sections split by ---.
.pptSlide titles → ##; body placeholders → - item bullets (multi-line) or prose; freeform text boxes included; slides split by ---.
.pdfHeadings inferred from font size vs document average → # / ## / ###; bullet lists normalized to - item; tab/space-aligned tables → pipe tables; page boundaries → ---.
.html, .htmH1–H6 → #######; ordered/unordered lists; fenced code blocks; pipe tables with an auto-detected header row; | in cells escaped.
.xlsx, .xlsEach non-empty sheet → ## SheetName + a pipe table; sheets split by ---.
.csv, .tsvA single pipe table; first row = header with a | --- | separator; delimiter auto-detected or set explicitly.
.md, .txtReturned as-is (no transformation).

With list_images=True, image-bearing formats additionally emit ![](hash.ext) references at the natural anchor point (top of each PDF page, after each Excel sheet, at the anchoring DOCX/DOC paragraph, inside the owning slide) and return the raw bytes in MarkdownResult.images.

Image extraction

list_images=True extracts embedded images for: DOC, DOCX (family), PPT, PPTX (family), XLSX (family — incl. ODS/XLSB, not XLS), HTML/HTM, PDF, EPUB, IPYNB, EML/MBOX/MSG, ODT/ODP.

Formats without image support: CSV, TSV, TXT, MD, JSON/JSONL/NDJSON, RTF, XLS.

.msg does extract images

Outlook .msg attachments are detected by filename, then MIME type, then magic bytes — Outlook frequently stores a real JPEG as application/octet-stream, so filename alone is not enough.

Only web-renderable formats are extracted (.png, .jpg, .jpeg, .gif, .webp); vector/metafile payloads (.emf, .wmf, .pict, .dib, .tiff) are silently skipped. Most formats key images by a stable content hash, so a picture that appears several times is stored once but referenced at each occurrence. PDF is the exception: its images are named positionally (image_p3_1.png) and always re-encoded to .png; EPUB, IPYNB and the email formats keep the source filename. Image extraction is batch-only — see Streaming. Full per-format key lists are in the Metadata Reference.

Format-specific notes

A few behaviors are worth knowing before you rely on a format:

  • PDF default vs structural differ (they are not aliases): default ranks heading sizes within each page and holds one page at a time; structural ranks across the whole document for a consistent hierarchy. Same text, different heading calls — see structural. A scanned / image-only PDF (no text layer) raises for text modes but succeeds under list_images=True, returning one image per page — the page's own embedded scan where it has one, a render otherwise. See Error Handling for the naming and the size caveat.
  • DOC / PPT (97–2003) are parsed by a pure-Rust binary reader (no LibreOffice, no external process). Only Word 97+ .doc is supported — pre-Word 97 files raise. .doc chunks carry the same structural metadata .docx does — section_heading, heading_path, list_level and table_rows/table_columns/table_cells — read from the file's own paragraph properties. .doc page_number counts the hard page breaks the file declares and is null when it declares none: Word recomputes soft pagination at render time and does not store it.
  • PPT reads slides from the presentation's slide list in order; speaker notes and master text are excluded, and slides are separated by ---.
  • XLS vs XLSX: .xls cannot expose named tables or print areas, so table mode falls back to bounding-box detection and page_aware to the full sheet; .xls also returns no images. All other behavior matches .xlsx. See the Metadata Reference.
  • Header detection (spreadsheets/CSV): the first all-string row of each sheet is auto-detected as the header and excluded from chunk content; unlabeled columns become Column 1, Column 2, ….

On this page