JavaScript — js-chunks

npm install js-chunks

js-chunks is the engine compiled to WebAssembly with a TypeScript wrapper. The package is ESM-only ("type": "module", "sideEffects": false); a require() of it needs Node ≥ 20.19. Declared engine floor is Node ≥ 18. It runs on Node, Bun, Deno and in browsers.

Functions

getChunks

function getChunks(
  source: ChunkSource,
  opts?: ChunkOptions & { listImages?: false },
): Promise<Chunk[]>;

function getChunks(
  source: ChunkSource,
  opts: ChunkOptions & { listImages: true },
): Promise<ChunksWithImages>;

getMarkdown

function getMarkdown(
  source: ChunkSource,
  opts?: ChunkOptions & { listImages?: false },
): Promise<string>;

function getMarkdown(
  source: ChunkSource,
  opts: ChunkOptions & { listImages: true },
): Promise<MarkdownWithImages>;

streamChunks

function streamChunks(
  source: ChunkSource,
  opts?: ChunkOptions,
): AsyncGenerator<Chunk, void, unknown>;

fitTokens

function fitTokens(
  chunks: Iterable<Chunk>,
  counter: TokenCounter,
  budget: number,
  opts?: FitOptions,
): Chunk[];

Takes getChunks output and re-fits it so no chunk exceeds budget under your tokenizer. counter is any (text: string) => number — gpt-tokenizer, a transformers.js tokenizer, or something you wrote.

import { getChunks, fitTokens } from "js-chunks";
import { encode } from "gpt-tokenizer";

const chunks = await getChunks("report.pdf");
const fitted = fitTokens(chunks, (s) => encode(s).length, 512);

The chunks keep their contentType and metadata, including the repeated table headers that make a retrieved row interpretable — which is the difference between this and running a text splitter over the raw document.

This is the only synchronous export in the package: pure TypeScript, no wasm, no I/O, so it behaves identically in Node, Bun, Deno and the browser.

The budget holds on every input, including whitespace-free text — CJK prose, long URLs, base64, minified code. The split ladder is paragraph → sentence → whitespace → a character-level bisection sized by your own counter, so an unbreakable-looking run is split at the character level rather than silently emitted over budget (before 0.6.4 it leaked through whole). Parts reassemble losslessly.

Pass overlap to share tokens between the parts of a chunk that had to be split — an integer token count, or a fraction of the budget in (0, 1):

const fitted = fitTokens(chunks, counter, 512, { overlap: 64 });

Overlap is scoped within one engine chunk on purpose (overlapping across chunks would smear contentType and structural metadata), and overlapped parts still respect the budget — the shared tail is trimmed until the joined part fits, because tokenisation is not additive across a join.

Split chunks gain fit_part / fit_total in their metadata. The input is never mutated. Bad arguments — a budget below 1, a negative minTokens, a non-function counter, an overlap at or above the budget, or an unrecognised split / merge / mergeMetadata / oversize — throw ChunkError with kind: "invalid-arg".

chunkText

function chunkText(text: string, opts?: ChunkOptions): Promise<Chunk[]>;

Chunk a bare string — no file, no extension. The text goes through the same plain-text pipeline a .txt file would, so it returns the usual structure-aware chunks and composes with fitTokens:

import { chunkText, fitTokens } from "js-chunks";

const chunks = await chunkText(rawText);
const fitted = fitTokens(chunks, counter, 512);

Strings are content here, never paths — pass a path to getChunks instead. Anything that is not a string throws ChunkError with kind: "invalid-arg".

The host-parsed PDF family

For callers who already parsed a PDF with some other tool and want chunk-engine's chunking over that Markdown. .pdf sources are parsed by the engine itself — you do not need these to chunk a PDF.

function chunkPdfMarkdown(
  markdown: string,
  totalPages: number,
  opts?: ChunkOptions,
): Promise<Chunk[]>;

function chunkPdfMarkdownWithImages(
  markdown: string,
  images: ChunkImage[],
  totalPages: number,
  opts?: ChunkOptions,
): Promise<ChunksWithImages>;

function normalizePdfMarkdown(markdown: string): Promise<string>;

chunkPdfMarkdownWithImages expects each supplied image's name to match an ![](name) reference in the Markdown. normalizePdfMarkdown returns just the normalised Markdown string — what getMarkdown would have emitted.

Types

interface Chunk {
  content: string;
  contentType: string;
  metadata: Record<string, unknown>;
}

interface ChunkImage {
  name: string;
  data: Uint8Array;
}

interface ChunksWithImages {
  chunks: Chunk[];
  images: ChunkImage[];
}

interface MarkdownWithImages {
  markdown: string;
  images: ChunkImage[];
}

ChunkOptions

interface ChunkOptions {
  mode?: ChunkMode;            // default "default"
  windowSize?: number;         // default 3
  overlap?: number;            // default 1
  sentencesPerChunk?: number;  // default 3
  paragraphsPerPage?: number;  // default 15
  /** Required when `source` is raw bytes and carries no name of its own. */
  filename?: string;
  /** Return extracted embedded images alongside the result. */
  listImages?: boolean;
}

There is no JavaScript equivalent of py-chunks' delimiter / encoding / rows_per_chunk arguments — CSV and spreadsheet knobs are not exposed through this surface.

TokenCounter and FitOptions

type TokenCounter = (text: string) => number;

interface FitOptions {
  /** Merge chunks below this many tokens into the next one. `0` disables. */
  minTokens?: number;
  merge?: "forward" | "none";
  split?: "sentence" | "paragraph" | "hard";
  mergeMetadata?: "first" | "union";
  oversize?: "split" | "keep" | "error";
  /** Never merge across a change in `boundaryKeys`. Default `true`. */
  respectBoundaries?: boolean;
  /** Default `["section_heading", "page_number", "sheet_name"]`. */
  boundaryKeys?: readonly string[];
  /** Tokens shared between split parts: an int, or a fraction of budget. */
  overlap?: number;
}

respectBoundaries is on by default because merging across a section, page or sheet makes the surviving metadata a lie — and it is the fastest way to destroy the table-header repetition that makes a retrieved row interpretable. mergeMetadata: "first" keeps the leading chunk's values and never invents one; "union" collects differing values into an array, so a chunk spanning two pages reports both rather than claiming one.

ChunkMode

All ten values, exactly as the union declares them:

type ChunkMode =
  | "default"
  | "section"
  | "semantic"
  | "sentence"
  | "page_aware"
  | "sliding_window"
  | "row"
  | "table"
  | "sheet"
  | "structural";

ByteSource and ChunkSource

type ByteSource = Uint8Array | ArrayBuffer | Blob;
type ChunkSource = string | ByteSource;

A string is a filesystem path, and only on Node — passing one elsewhere throws. Node's Buffer is not a separate arm because a Buffer is a Uint8Array. A File is a Blob, so it works and supplies its own name; an unnamed Blob needs opts.filename.

Errors

type ChunkErrorKind = "unsupported" | "invalid-arg" | "parse" | "io" | "unknown";

class ChunkError extends Error {
  readonly kind: ChunkErrorKind;
}

ChunkError covers every failure the engine raises and every argument validation the wrapper performs:

try {
  const chunks = await getChunks("./report.pdf");
} catch (e) {
  if (e instanceof ChunkError) {
    console.error(e.kind, e.message);   // "io" for a missing path
  } else {
    throw e;   // still worth keeping: an unexpected error should not be swallowed
  }
}

For parse failures raised by the engine, message is the engine's own message, byte-for-byte what py-chunks raises for the same input; the variant that Python expresses as an exception type is carried on kind instead.

message is not a cross-SDK contract everywhere, though. Two messages are built in each SDK's own host layer and differ in wording: the unsupported-extension message and the invalid-mode message. The numeric-range sentences (window_size, sentences_per_chunk, paragraphs_per_page) do agree now — every host layer was aligned on the engine's own wording, across both SDKs and across format families. kind agrees with Python's exception type in every case — branch on kind, log message. See Error Handling.

The wrapper's own type checks are the one place a message is deliberately not the engine's: they name the camelCase key you typed (windowSize must be an integer, got 2.5.), because a usize boundary has no counterpart for them. Out-of-range values reuse the engine's snake_case sentence instead, so the text is the same whichever side rejects.

Entry points

{
  ".": { "types": "./dist/index.d.ts", "import": "./dist/index.js" },
  "./web": { "types": "./pkg-web/chunks_wasm.d.ts", "default": "./pkg-web/chunks_wasm.js" },
  "./web/chunks_wasm_bg.wasm": "./pkg-web/chunks_wasm_bg.wasm"
}

The default entry (js-chunks) loads the WASM lazily on the first call and caches it. On Node it requires the synchronous pkg-node build — no async init step. Elsewhere it dynamically imports the pkg-web build and awaits its initialiser for you. This is the entry the API above documents.

js-chunks/web is the raw wasm-bindgen surface for bundled browser apps (Vite, webpack), where a dynamic import of a relative .js beside a .wasm does not survive bundling. You supply the .wasm URL and await the init yourself, once:

import initWasm, * as engine from "js-chunks/web";
import wasmUrl from "js-chunks/web/chunks_wasm_bg.wasm?url";

await initWasm({ module_or_path: wasmUrl });
const chunks = engine.getChunks(bytes, "report.docx", "default", 3, 1, 3, 15);

Note the shape: the /web functions are the positional wasm-bindgen exports, not the ergonomic wrapper — no options object, no overloads, and no ChunkError wrapping. See Installation.

One capability gap versus the native SDKs

PDF parsing runs in WASM — the same engine code as py-chunks and rs-chunks. What WASM cannot do is rasterise a page: the native builds fall back to PDFium rendering when a scanned PDF has no extractable text, and that fallback is compiled out of the WASM build. A text-less PDF therefore reports that it has no text here, instead of returning page images. It is the only deliberate behavioural divergence between the SDKs.

Related

• Input Sources — every accepted source, with examples.
• Framework Integration — Express, Fastify, Hono, Next.js.
• Output Schema — the shape of a Chunk.