# JavaScript — js-chunks

Every export from js-chunks, with exact TypeScript declarations, the /web subpath, and an accurate ChunkError contract.

```bash
npm install js-chunks
```

`js-chunks` is the engine compiled to WebAssembly with a TypeScript wrapper.
The package is **ESM-only** (`"type": "module"`, `"sideEffects": false`); a
`require()` of it needs Node ≥ 20.19. Declared engine floor is Node ≥ 18. It
runs on Node, Bun, Deno and in browsers.

## Functions

### getChunks

```ts
function getChunks(
  source: ChunkSource,
  opts?: ChunkOptions & { listImages?: false },
): Promise<Chunk[]>;

function getChunks(
  source: ChunkSource,
  opts: ChunkOptions & { listImages: true },
): Promise<ChunksWithImages>;
```

### getMarkdown

```ts
function getMarkdown(
  source: ChunkSource,
  opts?: ChunkOptions & { listImages?: false },
): Promise<string>;

function getMarkdown(
  source: ChunkSource,
  opts: ChunkOptions & { listImages: true },
): Promise<MarkdownWithImages>;
```

### streamChunks

```ts
function streamChunks(
  source: ChunkSource,
  opts?: ChunkOptions,
): AsyncGenerator<Chunk, void, unknown>;
```

  The WASM boundary is a synchronous full parse. `streamChunks` calls
  `getChunks`, waits for the complete array, and *then* yields its elements —
  the whole document and the whole chunk list are in memory before the first
  `yield`. Results and peak memory are identical to `getChunks`. Use it for
  `for await` ergonomics and to interleave downstream work with iteration, not
  to bound memory. `rs-chunks` and `py-chunks` do have truly incremental
  streaming for some formats; see [Streaming](/docs/streaming).

### fitTokens

```ts
function fitTokens(
  chunks: Iterable<Chunk>,
  counter: TokenCounter,
  budget: number,
  opts?: FitOptions,
): Chunk[];
```

Takes `getChunks` output and re-fits it so **no chunk exceeds `budget`** under
*your* tokenizer. `counter` is any `(text: string) => number` — `gpt-tokenizer`,
a transformers.js tokenizer, or something you wrote.

```ts

const chunks = await getChunks("report.pdf");
const fitted = fitTokens(chunks, (s) => encode(s).length, 512);
```

The chunks keep their `contentType` and `metadata`, including the repeated table
headers that make a retrieved row interpretable — which is the difference
between this and running a text splitter over the raw document.

This is the **only synchronous export** in the package: pure TypeScript, no
wasm, no I/O, so it behaves identically in Node, Bun, Deno and the browser.

  Every other API in this library produces byte-identical output across
  py-chunks, js-chunks and rs-chunks. **`fitTokens` does not, and cannot.** Its
  output depends on the token counter you pass, and tokenizers differ between
  languages — so putting a token budget in the engine would either vendor the
  wrong tokenizer for your embedding model or quietly break the byte-identical
  guarantee. Characters are parity-safe; tokens are not.

  py-chunks ships the same helper as
  [`fit_tokens`](/docs/api-reference/python#fit_tokens--fit-chunks-to-a-token-budget)
  — a separate implementation with the same behaviour, not a shared one.

The budget holds on **every** input, including whitespace-free text — CJK
prose, long URLs, base64, minified code. The split ladder is paragraph →
sentence → whitespace → a character-level bisection sized by your own counter,
so an unbreakable-looking run is split at the character level rather than
silently emitted over budget (before 0.6.4 it leaked through whole). Parts
reassemble losslessly.

Pass `overlap` to share tokens between the parts of a chunk that had to be
split — an integer token count, or a fraction of the budget in `(0, 1)`:

```ts
const fitted = fitTokens(chunks, counter, 512, { overlap: 64 });
```

Overlap is scoped *within* one engine chunk on purpose (overlapping across
chunks would smear `contentType` and structural metadata), and overlapped parts
still respect the budget — the shared tail is trimmed until the joined part
fits, because tokenisation is not additive across a join.

Split chunks gain `fit_part` / `fit_total` in their metadata. The input is never
mutated. Bad arguments — a budget below 1, a negative `minTokens`, a
non-function `counter`, an `overlap` at or above the budget, or an unrecognised
`split` / `merge` / `mergeMetadata` / `oversize` — throw `ChunkError` with
`kind: "invalid-arg"`.

  It is tempting to raise `minTokens` to pack chunks fuller. Measured on this
  project's own corpus, raising budget utilisation from 46% to 95% **halved**
  token-level precision with no gain in answerability. Use `minTokens` to remove
  genuinely useless fragments, not to fill the budget.

### chunkText

```ts
function chunkText(text: string, opts?: ChunkOptions): Promise<Chunk[]>;
```

Chunk a bare string — no file, no extension. The text goes through the same
plain-text pipeline a `.txt` file would, so it returns the usual
structure-aware chunks and composes with `fitTokens`:

```ts

const chunks = await chunkText(rawText);
const fitted = fitTokens(chunks, counter, 512);
```

Strings are content here, never paths — pass a path to `getChunks` instead.
Anything that is not a string throws `ChunkError` with `kind: "invalid-arg"`.

### The host-parsed PDF family

For callers who already parsed a PDF with some other tool and want
chunk-engine's chunking over that Markdown. `.pdf` *sources* are parsed by the
engine itself — you do not need these to chunk a PDF.

```ts
function chunkPdfMarkdown(
  markdown: string,
  totalPages: number,
  opts?: ChunkOptions,
): Promise<Chunk[]>;

function chunkPdfMarkdownWithImages(
  markdown: string,
  images: ChunkImage[],
  totalPages: number,
  opts?: ChunkOptions,
): Promise<ChunksWithImages>;

function normalizePdfMarkdown(markdown: string): Promise<string>;
```

`chunkPdfMarkdownWithImages` expects each supplied image's `name` to match an
`![](name)` reference in the Markdown. `normalizePdfMarkdown` returns just the
normalised Markdown string — what `getMarkdown` would have emitted.

## Types

```ts
interface Chunk {
  content: string;
  contentType: string;
  metadata: Record<string, unknown>;
}

interface ChunkImage {
  name: string;
  data: Uint8Array;
}

interface ChunksWithImages {
  chunks: Chunk[];
  images: ChunkImage[];
}

interface MarkdownWithImages {
  markdown: string;
  images: ChunkImage[];
}
```

  `contentType` is camelCase here — the WASM core emits `content_type`, and the
  wrapper renames exactly that one field. Keys **inside** `metadata` are not
  renamed: they stay snake_case, identical to py-chunks and rs-chunks. And
  `images` is an **array** here, where py-chunks returns a `dict`.

### ChunkOptions

```ts
interface ChunkOptions {
  mode?: ChunkMode;            // default "default"
  windowSize?: number;         // default 3
  overlap?: number;            // default 1
  sentencesPerChunk?: number;  // default 3
  paragraphsPerPage?: number;  // default 15
  /** Required when `source` is raw bytes and carries no name of its own. */
  filename?: string;
  /** Return extracted embedded images alongside the result. */
  listImages?: boolean;
}
```

There is no JavaScript equivalent of py-chunks' `delimiter` / `encoding` /
`rows_per_chunk` arguments — CSV and spreadsheet knobs are not exposed through
this surface.

### TokenCounter and FitOptions

```ts
type TokenCounter = (text: string) => number;

interface FitOptions {
  /** Merge chunks below this many tokens into the next one. `0` disables. */
  minTokens?: number;
  merge?: "forward" | "none";
  split?: "sentence" | "paragraph" | "hard";
  mergeMetadata?: "first" | "union";
  oversize?: "split" | "keep" | "error";
  /** Never merge across a change in `boundaryKeys`. Default `true`. */
  respectBoundaries?: boolean;
  /** Default `["section_heading", "page_number", "sheet_name"]`. */
  boundaryKeys?: readonly string[];
  /** Tokens shared between split parts: an int, or a fraction of budget. */
  overlap?: number;
}
```

`respectBoundaries` is on by default because merging across a section, page or
sheet makes the surviving metadata a lie — and it is the fastest way to destroy
the table-header repetition that makes a retrieved row interpretable.
`mergeMetadata: "first"` keeps the leading chunk's values and never invents one;
`"union"` collects differing values into an array, so a chunk spanning two pages
reports both rather than claiming one.

### ChunkMode

All ten values, exactly as the union declares them:

```ts
type ChunkMode =
  | "default"
  | "section"
  | "semantic"
  | "sentence"
  | "page_aware"
  | "sliding_window"
  | "row"
  | "table"
  | "sheet"
  | "structural";
```

### ByteSource and ChunkSource

```ts
type ByteSource = Uint8Array | ArrayBuffer | Blob;
type ChunkSource = string | ByteSource;
```

A `string` is a **filesystem path**, and only on Node — passing one elsewhere
throws. Node's `Buffer` is not a separate arm because a `Buffer` *is* a
`Uint8Array`. A `File` is a `Blob`, so it works and supplies its own `name`;
an unnamed `Blob` needs `opts.filename`.

## Errors

```ts
type ChunkErrorKind = "unsupported" | "invalid-arg" | "parse" | "io" | "unknown";

class ChunkError extends Error {
  readonly kind: ChunkErrorKind;
}
```

`ChunkError` covers **every failure the engine raises and every argument
validation the wrapper performs**:

| Failure | `kind` |
| --- | --- |
| Unsupported extension | `"unsupported"` |
| Unknown mode, bad parameter | `"invalid-arg"` |
| A filesystem path passed off Node | `"invalid-arg"` |
| A byte source with no resolvable filename | `"invalid-arg"` |
| A `source` of an unsupported type | `"invalid-arg"` |
| `opts` is not an object, or a getter on it throws | `"invalid-arg"` |
| `filename` / `mode` is not a string; a count is not an integer, is `NaN`, or is negative | `"invalid-arg"` |
| The document failed to parse | `"parse"` |
| An I/O failure inside the engine | `"io"` |
| A path that cannot be read (missing, `EACCES`, `EISDIR`) | `"io"` |
| A `Blob` whose `arrayBuffer()` rejects | `"io"` |
| A thrown value carrying no recognised variant tag | `"unknown"` |

  Reading a **path** on Node goes through `fs.readFileSync` in the wrapper,
  before the engine is reached — and that read is now wrapped too. A missing
  file gives you a `ChunkError` with `kind: "io"` whose `message` is Node's own
  (`ENOENT: no such file or directory, open './report.pdf'`). There is no input
  for which these functions throw something that is not a `ChunkError`.

  ```ts
  try {
    const chunks = await getChunks("./report.pdf");
  } catch (e) {
    if (e instanceof ChunkError) {
      console.error(e.kind, e.message);   // "io" for a missing path
    } else {
      throw e;   // still worth keeping: an unexpected error should not be swallowed
    }
  }
  ```

For **parse** failures raised by the engine, `message` is the engine's own
message, byte-for-byte what py-chunks raises for the same input; the variant that
Python expresses as an *exception type* is carried on `kind` instead.

`message` is **not** a cross-SDK contract everywhere, though. Two messages are
built in each SDK's own host layer and differ in wording: the
unsupported-extension message and the invalid-mode message. The numeric-range
sentences (`window_size`, `sentences_per_chunk`, `paragraphs_per_page`) *do*
agree now — every host layer was aligned on the engine's own wording, across
both SDKs and across format families. `kind` agrees with Python's exception
type in every case — **branch on `kind`, log `message`.** See
[Error Handling](/docs/error-handling).

The wrapper's own type checks are the one place a message is deliberately
*not* the engine's: they name the **camelCase** key you typed
(`windowSize must be an integer, got 2.5.`), because a `usize` boundary has no
counterpart for them. Out-of-range values reuse the engine's snake_case
sentence instead, so the text is the same whichever side rejects.

## Entry points

```json
{
  ".": { "types": "./dist/index.d.ts", "import": "./dist/index.js" },
  "./web": { "types": "./pkg-web/chunks_wasm.d.ts", "default": "./pkg-web/chunks_wasm.js" },
  "./web/chunks_wasm_bg.wasm": "./pkg-web/chunks_wasm_bg.wasm"
}
```

**The default entry (`js-chunks`)** loads the WASM lazily on the first call and
caches it. On Node it requires the synchronous `pkg-node` build — no async init
step. Elsewhere it dynamically imports the `pkg-web` build and awaits its
initialiser for you. This is the entry the API above documents.

**`js-chunks/web`** is the raw wasm-bindgen surface for bundled browser apps
(Vite, webpack), where a dynamic import of a relative `.js` beside a `.wasm`
does not survive bundling. You supply the `.wasm` URL and await the init
yourself, once:

```ts

await initWasm({ module_or_path: wasmUrl });
const chunks = engine.getChunks(bytes, "report.docx", "default", 3, 1, 3, 15);
```

Note the shape: the `/web` functions are the positional wasm-bindgen exports,
not the ergonomic wrapper — no options object, no overloads, and no
`ChunkError` wrapping. See
[Installation](/docs/installation#bundlers-vitewebpack).

## One capability gap versus the native SDKs

PDF *parsing* runs in WASM — the same engine code as py-chunks and rs-chunks.
What WASM cannot do is *rasterise* a page: the native builds fall back to
PDFium rendering when a scanned PDF has no extractable text, and that fallback
is compiled out of the WASM build. A text-less PDF therefore reports that it
has no text here, instead of returning page images. It is the only deliberate
behavioural divergence between the SDKs.

## Related

- [Input Sources](/docs/input-sources) — every accepted `source`, with examples.
- [Framework Integration](/docs/framework-integration/javascript) — Express, Fastify, Hono, Next.js.
- [Output Schema](/docs/output-schema) — the shape of a `Chunk`.