chunk-engine

Format examples

What the extended formats actually produce — real chunks from the engine for JSON, email, RTF, EPUB, notebooks, and OpenDocument.

View raw

The core formats (PDF, Word, PowerPoint, Excel, HTML, Markdown, TXT, CSV) are covered throughout the docs. This page shows what the extended formats produce — every example below is real output from the engine, not illustration.

These formats route through the same get_chunks / getChunks API and the same chunking modes and output schema as everything else — you don't call anything special. Dispatch is by extension.

Structured data — JSON / JSONL / NDJSON

Each record is rendered as a key: value block and emitted as a bullet_list chunk; nested objects are indented. For JSONL / NDJSON you get one chunk per line; for JSON you get one chunk per top-level array element (or the whole document when it isn't an array).

// one line of an NDJSON access log
{"agent":"Debian APT-HTTP/1.3","bytes":0,"remote_ip":"93.180.71.3","request":"GET /downloads/product_1 HTTP/1.1","response":304}
// → one chunk, content_type "bullet_list"
agent: Debian APT-HTTP/1.3
bytes: 0
remote_ip: 93.180.71.3
request: GET /downloads/product_1 HTTP/1.1
response: 304

Metadata: section_heading, section_level, document_metadata. A 50k-line log becomes 50k independently-embeddable chunks — pair it with streaming to keep memory flat.

Email — EML / MSG / MBOX

The message subject becomes a heading chunk and the body becomes plain_paragraph chunk(s); the parser is encoding- and MIME-aware.

  • .eml (MIME) and .msg (Outlook) — a single message → subject heading + body paragraphs.
  • .mbox — a whole mailbox: a leading Mailbox — N messages heading, then the heading/paragraph/list/table chunks for every message in order.
// .eml → 2 chunks
[heading]          This is a test message
[plain_paragraph]  <the message body…>

Rich text — RTF

Headings and paragraphs, with correct decoding across the code pages RTF uses (cp1252 / Shift-JIS / etc.). A Japanese .rtf titled タイトル yields a heading chunk with the text intact — no mojibake.

eBooks — EPUB

Chunked per spine item (reading order), with richer provenance than most formats:

// heading chunk from an EPUB
content: "Loomings"
metadata: {
  spine_index: 0,
  href: "OPS/content_001.xhtml",
  heading_path: null,
  page_number: null,
  ...
}

The spine_index and href tell you exactly which document in the book each chunk came from — useful for deep-linking back into the source.

Notebooks — IPYNB

Markdown cells become prose chunks; code cells (and their outputs) become code_block chunks, fenced and kept whole.

# a code cell
print("Hello World")
// → content_type "code_block"
```print("Hello World")```

OpenDocument — ODT / ODP / ODS

The OpenDocument family mirrors its OOXML counterpart exactly — same content types, same metadata, same modes:

FormatBehaves likeEmits
.odt (text)DOCXheading, plain_paragraph, table, lists — the 7 document modes
.odp (presentation)PPTXheading (Slide N), plain_paragraph, bullet_list — one unit per slide
.ods (spreadsheet)XLSXrow_document + the 6 spreadsheet modes, with sheet_name / sheet_index / header_row metadata

Image extraction

EPUB, IPYNB, EML/MBOX, and ODT/ODP also support list_images=True. ODS follows the XLSX family. See Supported Formats.

On this page