# Input Sources

Every way to hand a document to chunk-engine — path, bytes, file object, upload, URL, Blob — and which of them your SDK actually has.

All three SDKs dispatch by **file extension**. A path carries its own extension;
every byte source needs a filename so the engine knows what it is looking at.
Nothing else about the source changes the output — the same bytes produce the
same chunks however they arrive.

## What each SDK accepts

One table, all three SDKs. "Batch" is the whole-result call; "Stream" is the
one-chunk-at-a-time iterator.

| Source | Python | JavaScript | Rust |
| --- | --- | --- | --- |
| Filesystem path | batch + stream | batch + stream (Node/Bun only) | batch + stream |
| Raw bytes | batch + stream | batch + stream | batch + per-format stream (PDF, XLSX) |
| File-like object | batch + stream | — | — |
| Framework upload | batch + stream | — | — |
| HTTP(S) / pre-signed URL | batch + stream | — | — |
| `Blob` / `File` | — | batch + stream | — |
| Markdown string (host-parsed PDF) | — | batch | — |

Python has the widest surface because it is the SDK that sits behind web
frameworks; JavaScript and Rust expect you to produce the bytes and hand them
over. Neither of those is a capability gap in the engine — every source below
ends in the same two engine calls.

**Python**

Python's source-agnostic surface is **13 functions**: `get_chunks` and five
`get_chunks_from_*`, `stream_chunks` and five `stream_chunks_from_*`, plus
`get_markdown`. The `get_chunks` / `stream_chunks` pair sniffs the source and
forwards to the specific one, so you rarely need the others by name — but they
exist, they are exported, and calling them directly skips the sniffing.

```python
from py_chunks import (
    get_chunks,                            # sniffs any of the five below
    get_chunks_from_path,
    get_chunks_from_bytes,
    get_chunks_from_fileobj,
    get_chunks_from_upload,
    get_chunks_from_s3_presigned_url,
    stream_chunks,                         # sniffs any of the five below
    stream_chunks_from_path,
    stream_chunks_from_bytes,
    stream_chunks_from_fileobj,
    stream_chunks_from_upload,
    stream_chunks_from_s3_presigned_url,
    get_markdown,
)
```

`get_chunks` picks a branch in this order: `str`/`os.PathLike` (an `http`/`https`
scheme routes to the URL helper, anything else to the path helper) →
`memoryview`/`bytearray`/`bytes` → an object with a `.filename` attribute
(upload) → an object with `.read()` (file-like) → `TypeError`. Note that the
upload check comes **before** the file-like check, so any object carrying a
`filename` attribute takes the upload path.

**JavaScript**

JavaScript has **one polymorphic function per operation**, not a family. Every
source goes through `getChunks` / `getMarkdown` / `streamChunks`, and the
accepted types are:

```ts
type ByteSource = Uint8Array | ArrayBuffer | Blob;
type ChunkSource = string | ByteSource;
```

A Node `Buffer` is a `Uint8Array` and a `File` is a `Blob`, so both are accepted
structurally. A `DataView`, a non-`Uint8Array` TypedArray, a Node stream or a
`ReadableStream` are **not** — convert them first.

**Rust**

Rust exposes **six source-agnostic functions**, all on the crate root
(`chunks_rs::`). Three take a path, three take bytes:

```rust
pub fn get_chunks(file_path: &str, mode: &str, window_size: usize, overlap: usize,
                  sentences_per_chunk: usize, paragraphs_per_page: usize)
    -> Result<Vec<Chunk>>;

pub fn get_chunks_from_bytes(data: &[u8], filename: &str, mode: &str, window_size: usize,
                             overlap: usize, sentences_per_chunk: usize,
                             paragraphs_per_page: usize)
    -> Result<Vec<Chunk>>;

pub fn get_chunks_with_images_from_bytes(data: &[u8], filename: &str, mode: &str,
                                         window_size: usize, overlap: usize,
                                         sentences_per_chunk: usize,
                                         paragraphs_per_page: usize)
    -> Result<(Vec<Chunk>, Vec<(String, Vec<u8>)>)>;

pub fn get_markdown(file_path: &str) -> Result<String>;
pub fn get_markdown_from_bytes(data: &[u8], filename: &str) -> Result<String>;
pub fn get_markdown_with_images_from_bytes(data: &[u8], filename: &str)
    -> Result<(String, Vec<(String, Vec<u8>)>)>;
```

Two asymmetries are worth internalising before you plan around them:

- **Image extraction at the crate root is bytes-only.** There is no
  `get_chunks_with_images` and no `get_markdown_with_images` taking a path —
  `std::fs::read` the file and use the `_from_bytes` variant, or drop to the
  format module (`formats::docx::chunk_with_images(path, …)`, available for 12
  formats).
- **There is no streaming at the dispatch layer at all.** Streaming is
  per-format: `formats::<fmt>::stream(path, …)`.

## Filesystem path

```python
from py_chunks import get_chunks, get_chunks_from_path

# get_chunks() sniffs the source type; get_chunks_from_path() is the explicit one.
chunks = get_chunks("notes.md")
chunks = get_chunks_from_path("notes.md", mode="section")

# A missing path raises FileNotFoundError before any parsing happens.
```

```ts
import { getChunks } from "js-chunks";

// String sources are read with node:fs — Node only. In a browser or Deno,
// pass bytes plus opts.filename instead (see the Blob / File samples).
const chunks = await getChunks("./notes.md", { mode: "section" });
```

```rust
use chunks_rs::get_chunks;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Positional: (path, mode, window_size, overlap, sentences_per_chunk,
    //              paragraphs_per_page). These are the defaults.
    let chunks = get_chunks("notes.md", "section", 3, 1, 3, 15)?;
    println!("{}", chunks.len());
    Ok(())
}
```

**Python**

`get_chunks_from_path` checks the file exists first and raises
`FileNotFoundError(f"File not found: {file_path}")` before any parsing happens.
`str` and `pathlib.Path` both work (anything implementing `os.PathLike` does).

Only `http` and `https` are recognised as URL schemes — an `s3://` or `file://`
string falls through to the path branch and fails as a missing file.

**JavaScript**

  A string source is read with `fs.readFileSync` reached through
  `createRequire`, gated on a runtime check that requires `process.versions.node`
  **and** the absence of a `Deno` global. So:

  - **Node** — works.
  - **Bun** — works (it reports `process.versions.node` and defines no `Deno`).
  - **Deno** — deliberately excluded, even though modern Deno could serve
    `node:fs`. A path throws
    `ChunkError { kind: "invalid-arg" }`: *"Filesystem paths are only supported
    on Node. Pass a Uint8Array/ArrayBuffer/Blob with opts.filename instead."*
  - **Browsers** — same error. Use a `Blob`, a `File`, or bytes.

  `opts.filename` overrides the path's basename when present, which is how you
  chunk `/tmp/upload-a91f` as a `.docx`.

**Rust**

`get_chunks` is the only path-based chunking entry point on the crate root, and
it takes no `filename` argument — the extension comes from the path. Windows
paths are fine.

## A bare string

New in **0.6.4**: text you already hold as a string chunks directly, with no
file, no extension, and no encoding dance.

```python
from py_chunks import chunk_text

chunks = chunk_text("raw text you already have in memory...")
```

```ts

const chunks = await chunkText(rawText);
```

The string is routed through the plain-text pipeline, so it returns the same
structure-aware chunks a `.txt` file would, takes the same mode parameters, and
composes with `fit_tokens` / `fitTokens`.

Strings stay unambiguous everywhere else: `get_chunks("...")` still treats a
string as a **path** and `chunk_text` treats it as **content** — one meaning
per function, so a filename-looking document can never be mistaken for a file
to open. In Rust, encode and use `get_chunks_from_bytes(text.as_bytes(),
"text.txt", …)`.

## Raw bytes

The common case: you already hold the document in memory and never want it on
disk.

```python
from py_chunks import get_chunks_from_bytes

with open("report.pdf", "rb") as f:
    data = f.read()

# Straight to the engine's no-filesystem API — nothing is written to disk.
# filename is used only for extension detection.
chunks = get_chunks_from_bytes(data, "report.pdf")
```

```ts
import fs from "node:fs";
import { getChunks } from "js-chunks";

const bytes = new Uint8Array(fs.readFileSync("report.pdf"));
const chunks = await getChunks(bytes, { filename: "report.pdf" });
```

```rust
use chunks_rs::get_chunks_from_bytes;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let data = std::fs::read("report.pdf")?;
    // (data, filename, mode, window_size, overlap, sentences_per_chunk,
    //  paragraphs_per_page)
    let chunks = get_chunks_from_bytes(&data, "report.pdf", "default", 3, 1, 3, 15)?;
    println!("{}", chunks.len());
    Ok(())
}
```

  Since **0.6.1**, byte sources go straight to the engine — no temporary file,
  no round-trip through disk. The filename you pass is used **only** to pick a
  parser by extension; nothing is ever written under that name.

  The one exception is **streaming from bytes**, which does write a temp file.
  See [Streaming from a non-path source](#streaming-from-a-non-path-source).

**Python**

`get_chunks(data, filename=...)` and `get_chunks_from_bytes(data, filename)` are
the same call; the latter makes `filename` positional and required.
`bytearray` and `memoryview` are converted for you.

Two errors to expect:

- `ValueError("filename is required when source is bytes")` — from `get_chunks`
  when you omit it.
- `ValueError("data is empty")` — zero-length input is rejected up front rather
  than parsed into nothing.

**JavaScript**

`Uint8Array`, `ArrayBuffer` and Node `Buffer` all work, and all of them require
`opts.filename` — none of them carries a name.

```ts
throw new ChunkError(
  "A filename is required for byte sources (pass opts.filename or a named Blob) " +
    "so the engine can route by extension.",
  "invalid-arg",
);
```

A `Uint8Array` is handed to WASM without being copied, so do not mutate the
buffer while the call is in flight.

**Rust**

`filename` is `&str` and only its extension matters — `"x.docx"` is as good as
`"/original/path/report.docx"`.

For images from bytes, the shape changes to a tuple and the images are
`(name, bytes)` pairs:

```rust
let (chunks, images) = chunks_rs::get_chunks_with_images_from_bytes(
    &data, "report.docx", "default", 3, 1, 3, 15,
)?;
```

## Blob and File

**JavaScript**

A `File` — from `<input type="file">`, a drag-drop handler, or a `FormData`
entry — carries `.name`, so the engine can route without help. A bare `Blob`
does not, and needs `opts.filename`.

```python
import io
from py_chunks import get_chunks_from_fileobj

# An open file carries its own .name — no filename argument needed.
with open("notes.md", "rb") as f:
    chunks = get_chunks_from_fileobj(f)

# A BytesIO does not, so name it. (.read() may return str or bytes; both work.)
buf = io.BytesIO(b"# Chunking Notes\n\nHello.")
chunks = get_chunks_from_fileobj(buf, filename="notes.md")
```

```ts
import { getChunks } from "js-chunks";

// A File (from <input type="file">) is a Blob that carries .name, so the
// engine can route without opts.filename.
const file = (document.querySelector("input[type=file]") as HTMLInputElement)
  .files![0];
const chunks = await getChunks(file);

// opts.filename still wins if you want to override the routing extension.
```

A bare `Blob` — from `fetch(...).blob()`, a canvas, a generated payload — has no
name at all:

```ts
import { getChunks } from "js-chunks";

// A plain Blob has no name, and the engine routes on the extension —
// so pass opts.filename. Works in the browser, Deno, Bun and Node.
const blob = await (await fetch("/notes.md")).blob();
const chunks = await getChunks(blob, { filename: "notes.md" });

// Without a filename this throws ChunkError { kind: "invalid-arg" }:
// "A filename is required for byte sources (pass opts.filename or a named Blob)
//  so the engine can route by extension."
```

`opts.filename` still wins over a `File`'s own name if you want to force the
routing extension. An empty-string name counts as no name.

**Python** / **Rust**

  `Blob` and `File` are JavaScript types — this section only applies to
  `js-chunks`. Switch the language selector above to see it.

  
**Python**

    The Python equivalent is a **file-like object** — see the next section.
  

  
**Rust**

    In Rust, read the bytes and use `get_chunks_from_bytes`.
  

## File-like objects

**Python**

`get_chunks_from_fileobj(file_obj, filename=None)` accepts anything with
`.read()` — an open file, a `BytesIO`, a `tempfile`, a socket-backed reader.

```python
import io
from py_chunks import get_chunks_from_fileobj

# An open file carries its own .name — no filename argument needed.
with open("notes.md", "rb") as f:
    chunks = get_chunks_from_fileobj(f)

# A BytesIO does not, so name it. (.read() may return str or bytes; both work.)
buf = io.BytesIO(b"# Chunking Notes\n\nHello.")
chunks = get_chunks_from_fileobj(buf, filename="notes.md")
```

```ts
import { getChunks } from "js-chunks";

// A File (from <input type="file">) is a Blob that carries .name, so the
// engine can route without opts.filename.
const file = (document.querySelector("input[type=file]") as HTMLInputElement)
  .files![0];
const chunks = await getChunks(file);

// opts.filename still wins if you want to override the routing extension.
```

Behaviour worth knowing:

- **Filename inference** is `filename or getattr(file_obj, "name", None)`. An
  `open()` handle supplies it; a `BytesIO` does not, so name it or you get
  `ValueError("filename is required when file object has no name")`.
- **Text mode works**: a `str` from `.read()` is re-encoded as UTF-8. That
  ignores the file's own encoding, so open binary files in `"rb"`.
- **The object is not rewound.** `.read()` is called once, with no `seek(0)`, so
  a stream you already consumed yields `b""` and then
  `ValueError("data is empty")`.

**JavaScript** / **Rust**

  Only `py-chunks` has a file-object entry point.

  
**JavaScript**

    In JavaScript, read the stream into a `Uint8Array` (or wrap it in a `Blob`)
    and pass `opts.filename`.
  

  
**Rust**

    In Rust, read the reader into a `Vec<u8>` and call `get_chunks_from_bytes`.
  

## Framework uploads

**Python**

`get_chunks_from_upload(upload_file)` takes a FastAPI / Starlette `UploadFile`,
a Django `UploadedFile`, or anything else with a `.filename` attribute.

```python
from fastapi import FastAPI, UploadFile
from py_chunks import get_chunks_from_upload

app = FastAPI()

@app.post("/chunk")
def chunk(file: UploadFile):
    # Reads upload_file.file (the SpooledTemporaryFile) when present, because
    # UploadFile.read() is a coroutine — passing the object itself is safe in a
    # sync handler. filename comes from upload_file.filename.
    return get_chunks_from_upload(file, mode="section")
```

  The helper reads `upload_file.file` — the underlying `SpooledTemporaryFile` —
  when it is present, and only falls back to `upload_file.read()` when it is
  not. That ordering is what makes it safe in a **sync** handler: FastAPI's
  `UploadFile.read()` is a coroutine, and awaiting it is not possible there.

  If the fallback does hit a coroutine it raises
  `TypeError("upload_file.read() is async; pass upload_file.file or use bytes API")`
  rather than failing obscurely. In an `async def` handler, either pass
  `file.file` explicitly or `await file.read()` yourself and use the bytes API.

The filename comes from `upload_file.filename`; a missing or empty one raises
`ValueError("upload_file.filename is required")`. See
[Framework Integration](/docs/framework-integration) for complete handlers.

**JavaScript** / **Rust**

  Only `py-chunks` has an upload entry point, because the upload objects it
  targets are Python web-framework types.

  
**JavaScript**

    In an Express or Hono handler, a `multipart/form-data` field arrives as a
    `File` (Web `FormData`) or a `Buffer` (multer) — both are already accepted
    sources. See [Framework Integration](/docs/framework-integration).
  

  
**Rust**

    In Axum, a `Multipart` field gives you `Bytes` and `file_name()` — feed both
    to `get_chunks_from_bytes`.
  

## URLs and pre-signed links

```python
from py_chunks import get_chunks, get_chunks_from_s3_presigned_url

# Downloads with urlopen, then chunks the bytes in memory (nothing hits disk).
chunks = get_chunks_from_s3_presigned_url(url, timeout=60)

# The filename defaults to the last URL path segment; override it when the
# URL has no useful name (pre-signed links often don't).
chunks = get_chunks_from_s3_presigned_url(url, filename="report.pdf")

# get_chunks() routes http/https sources here for you.
chunks = get_chunks(url, filename="report.pdf")
```

```ts
import { getChunks } from "js-chunks";

// js-chunks has no URL helper — fetch it yourself, then pass the bytes
// plus a filename so the engine can route by extension.
const res = await fetch(url);
const chunks = await getChunks(await res.arrayBuffer(), {
  filename: "report.pdf",
});
```

```rust
use chunks_rs::get_chunks_from_bytes;

// rs-chunks has no URL helper either: download with your HTTP client of
// choice (reqwest, ureq, …) and chunk the response body. `filename` is used
// only for extension routing — nothing is written under that name.
fn chunk_download(body: &[u8], filename: &str) -> Result<(), Box<dyn std::error::Error>> {
    let chunks = get_chunks_from_bytes(body, filename, "default", 3, 1, 3, 15)?;
    println!("{} chunks", chunks.len());
    Ok(())
}
```

**Python**

`get_chunks_from_s3_presigned_url(url, filename=None, timeout=60)` downloads
with `urllib.request.urlopen` and chunks the bytes in memory. `get_chunks` routes
any `http`/`https` string here automatically.

The filename defaults to the last path segment of the URL, with the query string
stripped — so an S3 signature does not end up looking like an extension. If the
path has no final segment you get
`ValueError("filename is required when URL path has no filename")`; pass
`filename=` for pre-signed links that carry an opaque key.

  `get_markdown` has a **narrower** source list than `get_chunks`, and its path
  branch is a literal `Path(source).is_file()` check. A URL string is not a
  file, so it raises `FileNotFoundError: File not found: https://…` — it does
  not download anything.

  `get_markdown` also has **no upload branch**: pass a FastAPI `UploadFile` and
  it falls through to the file-like branch, where the async `read()` fails
  unhelpfully. Pass `upload_file.file`, or read the bytes and call
  `get_markdown(data, filename="report.pdf")`.

  Accepted by `get_markdown`: `str` / `pathlib.Path` (existing file),
  `bytes` / `bytearray` / `memoryview` (with `filename=`), and any object with
  `.read()` (with `filename=` or a `.name`).

**JavaScript** / **Rust**

There is no URL helper — fetch the bytes yourself and pass them with a filename.
That is a deliberate omission, not a gap: your HTTP client already handles the
auth, retries, proxies and timeouts an in-library downloader would have to
reinvent.

## Streaming from a non-path source

```python
from py_chunks import stream_chunks_from_bytes

# The engine's streaming surface is path-based, so this one source *does* touch
# disk: the bytes go to a NamedTemporaryFile that is deleted when the iterator
# is exhausted, closed, or the with-block exits. (Batch bytes never touch disk.)
with stream_chunks_from_bytes(data, "report.pdf", mode="section") as chunks:
    for chunk in chunks:
        handle(chunk)
```

```ts
import { streamChunks } from "js-chunks";

// Bytes stream the same way paths do — but see the note in `streamingAdvanced`:
// in JS this is async-iteration ergonomics, not bounded memory.
for await (const chunk of streamChunks(bytes, { filename: "report.pdf" })) {
  handle(chunk);
}
```

```rust
use chunks_rs::formats::pdf;

// Only pdf and xlsx expose a bytes-streaming entry point; every other format
// streams from a path (`formats::<fmt>::stream`). There is no bytes streaming
// on the source-agnostic dispatch layer at all.
fn main() -> Result<(), Box<dyn std::error::Error>> {
    let data = std::fs::read("report.pdf")?;
    for chunk in pdf::stream_from_bytes(data, "section", 3, 1, 3, 15) {
        let chunk = chunk?;
        println!("{}", chunk.content_type);
    }
    Ok(())
}
```

**Python**

  The engine's streaming surface is path-based. `stream_chunks_from_bytes`
  therefore writes the bytes to a `NamedTemporaryFile` (suffixed with the real
  extension), builds the iterator over it, and deletes it again — on iterator
  exhaustion, on an exception during iteration, on an explicit `.close()`, on
  leaving a `with` block, and as a last resort in `__del__`.

  `stream_chunks_from_fileobj`, `stream_chunks_from_upload` and
  `stream_chunks_from_s3_presigned_url` all funnel through it, so **every
  non-path streaming source** materialises a temp file. Batch sources never do.

  Use `with` (the returned iterator is a context manager) or drain it fully; an
  abandoned half-read iterator only cleans up when the garbage collector gets to
  it.

`list_images` is a batch-only option — no `stream_*` function accepts it.

**JavaScript**

`streamChunks` accepts exactly the same sources as `getChunks`. Be clear-eyed
about what it buys you: the WASM boundary is a synchronous full parse, so the
whole chunk array exists before the first `yield`. It is async-iteration
ergonomics — useful for overlapping embedding work with iteration — not bounded
memory. `listImages` is forced to `false` inside it.

**Rust**

Streaming lives on the format modules. Seventeen of them expose
`stream(path, …)`; only **`pdf` and `xlsx`** additionally expose
`stream_from_bytes`. There is no bytes streaming on the dispatch layer.

Note that most prose formats' `stream` is an eager `chunk(...)` fed through
`into_iter()` — genuinely incremental backends exist for PDF (`"default"` mode),
CSV and XLSX. [Streaming](/docs/streaming) has the per-format truth.

## Markdown you already parsed

**JavaScript**

If another tool produced the Markdown — a hosted PDF service, a scraper, an OCR
pass — you can hand the string to the engine's chunker directly and skip parsing
entirely.

```ts
import {
  chunkPdfMarkdown,
  chunkPdfMarkdownWithImages,
  normalizePdfMarkdown,
} from "js-chunks";

// You already have Markdown from some other PDF parser and want the engine's
// chunking over it. (.pdf input is parsed by the engine itself — this is for
// callers who parsed it elsewhere. It is what the playground drives.)
// totalPages populates document_metadata.total_pages.
const chunks = await chunkPdfMarkdown(markdown, 15, { mode: "section" });

// With host-supplied images: name must match the ![](name) reference.
// Resolves to { chunks, images }, image chunks first.
const withImages = await chunkPdfMarkdownWithImages(markdown, images, 15);

// Just the engine's normalisation step, when you need the string itself.
const normalised = await normalizePdfMarkdown(markdown);
```

- `chunkPdfMarkdown(markdown, totalPages, opts?)` — `markdown` and `totalPages`
  are required positionals; `totalPages` populates
  `document_metadata.total_pages`.
- `chunkPdfMarkdownWithImages(markdown, images, totalPages, opts?)` — always
  resolves to `{ chunks, images }` regardless of `opts.listImages`, image chunks
  first. Each image's `name` must match its `![](name)` reference in the string.
- `normalizePdfMarkdown(markdown)` — the engine's normalisation step alone, when
  you want the cleaned string rather than chunks.

These are the calls the [playground](/playground) runs.

**Python** / **Rust**

  The Markdown-string entry points (`chunkPdfMarkdown`,
  `chunkPdfMarkdownWithImages`, `normalizePdfMarkdown`) are **JavaScript-only**.

  
**Python**

    In Python, write the Markdown to a `.md` file or pass it as bytes with
    `filename="doc.md"` — you get the Markdown chunking pipeline, minus the
    PDF-specific page metadata.
  

  
**Rust**

    In Rust, pass the string's bytes with a `.md` filename to
    `get_chunks_from_bytes` — you get the Markdown chunking pipeline, minus the
    PDF-specific page metadata.
  

## Bundled browsers

**JavaScript**

The default `js-chunks` entry resolves the browser WASM build through a dynamic

the raw wasm-bindgen surface — positional arguments, snake_case chunk fields —
and it accepts bytes only. See
[Installation → Bundlers](/docs/installation#bundlers-vitewebpack).

**Python** / **Rust**

  Browser packaging concerns `js-chunks` only.

## Next steps

- [Output Schema](/docs/output-schema) — what comes back, whatever went in
- [Streaming](/docs/streaming) — the honest per-format, per-runtime matrix
- [Error Handling](/docs/error-handling) — what each bad source raises
- [Framework Integration](/docs/framework-integration) — FastAPI, Express, Axum
- [API Reference](/docs/api-reference) — every signature, per language