semantic

semantic walks the document one block at a time and asks a single question at each step: does this block continue the one before it? If yes it is merged into the current chunk; if no, the chunk is closed and a new one starts.

There is no embedding model and no similarity score. The decision is a fixed ladder of lexical signals, evaluated in strict priority order, and the signal that fired is written into the chunk's metadata — so a boundary you disagree with is always explainable.

Real input → real output

from py_chunks import get_chunks

# Adjacent blocks merge when they look like the same idea. Every chunk records
# WHY it starts where it does, in primary_merge_reason (+ merge_reasons).
for c in get_chunks("notes.md", mode="semantic"):
    print(c["content_type"], "|", c["metadata"]["primary_merge_reason"])

# heading | initial
# semantic | initial
# heading | initial
# semantic | initial
# table | structural_boundary
# heading | initial
# semantic | initial

The signal ladder

For each candidate block, in this order. The first rule that matches decides.

Prefix matching is ASCII case-insensitive.

The emitted vocabulary

primary_merge_reason is always one of:

initial · reference_continuity · elaboration · example · cause_effect · contrast_continuation · question_answer · definition_expansion · short_paragraph · list_continuation · keyword_overlap · structural_boundary

Two of those never come from the ladder:

• initial — a chunk built from exactly one block. Nothing was merged, so there is no reason to report. Its merge_reasons array is empty.
• structural_boundary — the chunk is a code block or a table. Those are always emitted standalone and never participate in merging.

And one name from the ladder is never emitted: transition_break. It only ever ends a chunk, so it is never a reason a block joined one.

merge_reasons is the deduplicated list of every reason that fired inside the chunk, in the order they first fired, with initial excluded.

Deterministic tie-break

primary_merge_reason is the reason that fired the most times in that chunk. When two reasons tie, the winner is chosen by count descending, then reason name ascending — so elaboration beats keyword_overlap on a tie. This is a sort, not a hash-map iteration, which is why the three SDKs agree byte-for-byte on every chunk.

The key names differ by format

This is the part that bites cross-format code. Not every format writes the same merge keys:

DOCX and PPTX also use vocabulary the markdown ladder does not:

• DOCX adds docx_heading (chunk opened on a heading paragraph), heading_merge (a heading absorbed into the following body) and size_limit (the chunk was closed by the cap).
• PPTX adds single_unit (the chunk is one whole slide, nothing merged).

Read metadata.get("primary_merge_reason") or metadata.get("merge_reason") if you want one line that works everywhere.

Units and formats

What it emits

semantic mode does not emit only semantic chunks.

Measured: code_heavy.md → 13 semantic + 22 heading + 17 code_block + 2 table; all_round.docx → 13 semantic and nothing else. Lists are merged into semantic chunks — unlike in sentence mode, where they stay bullet_list.

Parameters

semantic reads none of the four mode parameters. The 1,500-character cap is a compile-time constant, not a knob.

Metadata

Beyond the merge keys, markdown-family semantic chunks carry heading_path, section_heading, section_level, block_types, paragraph_count, has_list, keyword_density, avg_block_length and chunk_index. Full listing in the Metadata Reference.

When to use it

• Feeding an embedding model — coherent chunks embed better.
• LLM context where you want complete thoughts, not fragments.
• General-purpose RAG when you're not sure which mode to pick.