Metadata Reference

metadata is a map whose keys depend on both the format and the mode. Output Schema explains the object; this page enumerates the keys.

Every table below was verified against live output at default parameters (window_size 3, overlap 1, sentences_per_chunk 3, paragraphs_per_page 15) over the project's fixture corpus.

Two chunker families

Which table applies to your file depends on which chunker handles it.

• The shared pipeline — PDF, Markdown, HTML, TXT, RTF, JSON/JSONL/NDJSON, EML/MBOX/MSG, ODT/ODP, EPUB, IPYNB. One implementation, so the per-mode key set is the same for all of them, plus a small set of format-specific additions.
• Native chunkers — DOCX, DOC, PPTX, PPT, and the spreadsheet / delimited families. Each has its own key vocabulary.

The shared pipeline

Core keys by mode

Emitted by every pipeline format in that mode — but not on every chunk. The derived modes pass some blocks through untouched, and a pass-through chunk carries the vocabulary of what it is rather than of the mode that produced it. The exceptions are worth knowing before you index on a key — they are enumerated and verified below:

Keys that are absent, not null

Verified over the fixture corpus at default parameters. In each case the key is missing from the map entirely, so chunk["metadata"]["actual_sentence_count"] raises KeyError — use .get().

Every other key in the table above was present on every chunk of every pipeline format tested. See sentence mode for the pass-through rule in context.

Two variants

semantic and sliding_window come in two flavours inside the pipeline:

Per-format additions

On top of the core, in all modes unless noted.

document_metadata by format

The one nested object. Its shape is a union, not a struct — read the keys you need with a default.

What these formats produce

Real output, so the keys above have something to attach to.

JSON / JSONL / NDJSON — one chunk per record, rendered as key: value lines with nested objects indented, content_type: "bullet_list":

// elastic_products.ndjson, default
{ "content": "description: Latest flagship smartphone with titanium design…\nprice: 999.99\nproduct_name: iPhone 15 Pro",
  "content_type": "bullet_list",
  "metadata": { "record_range": [0, 0], "section_heading": null, "section_level": 0,
                "document_metadata": { "source_type": "ndjson", "top_level": "lines",
                                       "record_count": 25, "envelope_key": null } } }

record_range is [first, last], inclusive and 0-based. Element-level modes (default, structural, sentence) give a point range like [3, 3]; grouping modes partition the file into contiguous non-overlapping ranges; sliding_window ranges overlap by the window's overlap. The chunk's own record count is last - first + 1 — there is deliberately no chunk-level record_count, because document_metadata.record_count already means the whole file. envelope_key names the array a document was unwrapped from ("features" for GeoJSON) and is null when the document is the record set. A 50k-line log becomes 50k independently embeddable chunks — pair it with streaming.

EML / MBOX / MSG — the subject becomes a heading chunk, the body plain_paragraph chunks; the parser is MIME- and encoding-aware.

cpython_msg_01_plain.eml → 2 chunks
  [heading]          This is a test message
  [plain_paragraph]  From: John X. Doe bbb@ddd.com / To: bbb@zzz.org / Date: … / Hi, …

mimekit_simple.mbox → 18 chunks
  [heading]          Mailbox — 3 messages          (no message_index)
  [heading]          Message 1                     message_index: 1
  [heading]          this'll probably break a lot of mime parsers
  [plain_paragraph]  Hello world. …

An .mbox is many messages in one file, so its identity keys sit on each chunk rather than in document_metadata — "which message is this from?" is a lookup, not a string-parse. in_reply_to and references are lists of message-ids, so a thread can be rebuilt without re-parsing the source.

RTF — headings come from paragraph styles, not inline markup, so a converter that flattens the style sheet yields prose chunks with no heading structure rather than guessed ones. Decoding is code-page correct (cp1252, Shift-JIS, …), character formatting survives into the markdown, and list markers are normalised to their kind (the glyph a writer painted for a bullet is only meaningful in its own symbol font). Underline has no markdown equivalent and is not represented.

// conv_libreoffice_heading123.rtf, default
{ "content": "First paragraph", "content_type": "heading",
  "metadata": { "section_heading": null, "section_level": 1,
                "document_metadata": { "source_type": "rtf",
                                       "title": null, "author": "Paolo Mottadelli" } } }

title and author are deliberately null rather than a guess when the writer stored the value as a \upr pair, whose ANSI half is a lossy ????? copy.

EPUB — chunked per spine item, in reading order.

// epubcheck_valid.epub, default
{ "content": "Loomings", "content_type": "heading",
  "metadata": { "spine_index": 0, "href": "OPS/content_001.xhtml", "is_navigation": false,
                "page_number": null, "footnotes_captions": [], "heading_path": null,
                "section_heading": null, "document_metadata": { "…": "see above" } } }

Dublin Core elements are repeatable and several fixtures use that — the plural keys carry every value, the singular ones keep the first. toc is the book's own table of contents as {title, href} in reading order, read from nav.xhtml (EPUB 3) or toc.ncx (EPUB 2), so you get chapter titles even when the XHTML uses no heading tags. is_navigation marks a chunk that came from the TOC page rather than the content — a link list chunks as ordinary prose otherwise and pollutes retrieval.

IPYNB — markdown cells become prose chunks; code cells and their outputs become separate code_block chunks, fenced and kept whole:

// nbc_helloworld.ipynb, default → 2 chunks
{ "content": "```\nprint(\"Hello World\")\n```", "content_type": "code_block",
  "metadata": { "section_heading": null, "section_level": 0,
                "document_metadata": { "source_type": "ipynb", "nbformat": "4.0",
                                       "kernel": null, "language": null,
                                       "cell_count": 1, "code_cell_count": 1,
                                       "markdown_cell_count": 0 } } }

ODT / ODP — the OpenDocument family mirrors its OOXML counterpart: .odt behaves like DOCX, .odp like PPTX (one heading chunk titled Slide N per slide, then its body), .ods follows the XLSX family below.

// odftoolkit_SlideTest1.odp, default
{ "content": "Slide 1", "content_type": "heading",
  "metadata": { "slide_number": 1, "slide_title": "This is the test documents for ODF toolkit",
                "section_heading": null, "section_level": 2,
                "document_metadata": { "source_type": "odp", "title": null,
                                       "creator": "hanbiao", "slide_count": 1 } } }

ODP has no title element the way PPTX does, so slide_title is the slide's first line of text — what a reader would call its title.

DOCX

DOC and PPT (97–2003)

Both legacy readers emit the same key set in every mode — the mode changes how text is grouped, not what is described.

.doc page_number counts the hard page breaks the file declares and is null when it declares none — Word recomputes soft pagination at render time and does not store it.

PPTX

Spreadsheets (XLSX / XLS)

Identical for the whole family — .xlsm, .xlsb, .xltx, .xltm and .ods included. No document_metadata; provenance is per chunk. chunk_index and skipped_sheets are present in every mode.

CSV / TSV

CSV has no way to declare whether its first row is a header, so the engine infers it: a non-numeric cell above a numeric column reads as a label, and in a textual column a first cell whose length differs from the rest reads as a label.

• has_header: true — header_row came from the file, and that row is not emitted as data.
• has_header: false — the file is headerless. Every row is emitted as data and header_row holds synthetic Column 1 … Column N labels.

A single-row file is always treated as data, never as a lone header, so it still produces a chunk.

Image chunks

With list_images on, content_type: "image" chunks carry a deliberately small and non-uniform key set. (Without list_images, an image chunk is a structural placeholder with no image_name at all — see Output Schema.)

Naming. DOCX, PPTX, XLSX, HTML, ODT/ODP and the legacy DOC/PPT path use a content hash — "<16 hex>.<ext>", FNV-1a 64-bit, pinned in the engine so the name is identical in Python, JavaScript and Rust. PDF is positional (image_p3_1.png) and always re-encoded to .png. EPUB, IPYNB and the email formats keep the source name (7433694763631080598_cover.jpg, output_image_1.png, msg1_image_1.gif, testPNG.png).

Enumerated values

primary_merge_reason (pipeline semantic, and PPTX) — the winning signal: keyword_overlap, short_paragraph, reference_continuity, elaboration, example, cause_effect, contrast_continuation, question_answer, definition_expansion, list_continuation, heading_merge, heading (HTML/EPUB), section_divider and single_unit (PPTX), plus two "nothing fired" values: structural_boundary and initial (a singleton chunk). merge_reasons holds every signal that fired and is empty for those two.

DOCX merge_reason (singular, DOCX only) — keyword_overlap, short_paragraph, heading_merge, size_limit, reference_continuity, list_continuation, docx_heading, transition_break. size_limit marks a chunk closed because the next paragraph would exceed the 1,500-character cap; list_continuation marks a run of list items held together as one unit.

page_break_type — heading_boundary, estimated, explicit, rendered, section (DOCX), slide_boundary (PPTX).

block_types — heading, paragraph, list, table, code_block.