Document Chunker for Retrieval

knowledge.chunk_document

ProductionBounded custom (Class B)Deterministicstrictfastv1.0.0

No network

Overview

Turn one document into the units you will embed and retrieve. You choose how to cut it — at token windows, at Markdown headings, at blank lines, or at headings first and then token windows inside any section that is still too long — and you get back each chunk with its text, its token count, the exact character range it came from, the heading trail above it and any metadata you attached. Ids are derived from the chunk's own content, so re-chunking a document you have not changed gives you the same ids and your vector store is not rewritten for nothing. Token counts come from the same tokenizer the rest of the platform uses, so a count here means the same thing as a count anywhere else. Nothing is embedded, no model is called and nothing is stored. Reach for text.chunk_by_tokens instead when you only need a string cut at token boundaries and do not care where the pieces came from.

A retrieved chunk is only useful if you can say where it came from and cite it in context, and a chunk id is only useful if it survives a re-run. So every chunk carries the character range it was cut from and the headings above it, and its id comes from its own content instead of its position.

Reliability

The same document, settings and tokenizer version always produce the same chunks and the same ids, and nothing here reads a clock or a random number. Token counts come from the pinned tokenizer library whose version comes back in the summary; a tokenizer upgrade can move a token window, which is why we tell you which version measured your document. Character offsets never depend on the tokenizer. Failures, none of which get better on their own: CHUNK_TEXT_INVALID means text was empty — send at least one character; CHUNK_TEXT_TOO_LARGE means it was longer than 2000000 characters — send the document in parts; CHUNK_LIMITS_INVALID means a value in limits was outside its range, minTokens or overlapTokens was not below maxTokens, limits carried a property we do not accept, or format, strategy or idMode named something that does not exist — correct the field named in the error and retry; CHUNK_LIMIT_EXCEEDED means your settings would produce more than 10000 chunks, which we cannot return at all — raise limits.maxTokens or send the document in parts; CHUNK_TOKENIZER_UNSUPPORTED means the encoding you named is not one we have, and the supported list comes back with the error; CHUNK_METADATA_INVALID means documentId was too long, or a metadata entry was too long, blank-keyed or not a scalar — fix that entry and retry; CHUNK_OUTPUT_TOO_LARGE means the chunks would carry more than 2000000 characters of text between them, which happens when overlap repeats a lot — lower limits.overlapTokens or send the document in parts.

Context footprint

Chunks carry their text, so the response is roughly the size of your document plus whatever overlap repeats, up to 2000000 characters in total and 10000 chunks. If that ceiling is in your way, lower overlapTokens first — it is usually what multiplied the response. A successful call costs 1 execution credit.

When to use it

Before you embed a document, when you want chunks that can be cited and re-chunked without churning your vector store.

When you only need a string cut at token boundaries, when you want the embeddings themselves, or when the document still has to be fetched.

Powered by JTokkit, an open-source tokenizer reproducing OpenAI tiktoken encodings, with Squillify's own boundary, heading and identity rules on top.

Scope

Note

Cut one supplied document into retrieval chunks carrying their own text, token count, character range, heading trail, metadata and a content-derived id.

Non-goals

Does not embed anything or call an embedding model
Does not fetch the document — you send the text
Does not store chunks, an index or a cache between calls
Does not summarise, clean or rewrite the text it returns
Does not parse Markdown beyond headings and fenced code blocks

Properties

IdempotentRetry safeStreaming supportedFile input supportedRequires external service

Capabilities

document-processingtext-processing

Cost & billing

Flat rate per invocation

MetercallsBillable onsuccessCost per call1 credit

File handling

Accepts file_refProduces file_refMax input2 MB

How we handle your data

Sensitive output possible

What this means

This skill may receive sensitive content and may return submitted, transformed, extracted, or reformatted content.

Squillify guarantees

Runs inside Squillify
This skill runs inside Squillify.
Payloads are not sent to an external service by this skill.
Squillify does not permanently store submitted payloads.
Telemetry records metadata only, not submitted content.
Submitted payloads are not used for training.
Submitted payloads are not reviewed by humans.
This skill does not target sensitive data.
The response may contain submitted or transformed content.

Your responsibility

Squillify does not determine whether submitted content is sensitive. Your application is responsible for deciding what data it is allowed to submit. Your application is responsible for securing the response.

Read the Microskill Data Handling Policy

Input schema

Field	Type	Required	Description
text	string	Yes	Required document to split, 1 to 2000000 Unicode code points; an empty string is refused.
limits	object	Yes	Closed size policy: required maxTokens plus optional minTokens, overlapTokens and maxChunks; any other property is refused.
format	string	No	PLAIN or MARKDOWN; omit it for PLAIN.
strategy	string	No	Where to cut the document; omit it for TOKEN_WINDOW.
tokenizer	string	No	Encoding used to count tokens; omit it for cl100k_base.
documentId	string	No	Up to 128 characters prefixed onto every chunk id; omit it for ids with no prefix.
metadata	object	No	Purpose-bound map of up to 32 scalar metadata entries copied onto every chunk; keys are 1 to 64 characters and values are a string, number or boolean. Omit it and every chunk carries an empty object.
idMode	string	No	How chunk ids are derived; omit it for CONTENT_HASH.

Output schema

Field	Type	Required	Description
chunks	array	Yes	One entry per chunk in document order, each a closed object with id, ordinal, text, tokenCount, charStart, charEnd, headingPath, metadata and flags.
summary	object	Yes	Closed run summary with chunkCount, totalTokens, applied strategy and tokenizer, truncation and front-matter detection.

API contract

POST /v1/execute

curl

curl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"knowledge.chunk_document","input":{"text":"# Refunds\n\nWe refund an unused subscription within 30 days.","format":"MARKDOWN","strategy":"HEADING_THEN_TOKEN","limits":{"maxTokens":256,"overlapTokens":32},"documentId":"refund-handbook"}}'

Integrations

TypeScript SDK

import { Squillify } from "@squillify/sdk";

const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });

const { output } = await sq.execute("knowledge.chunk_document", {"text":"---\ntitle: Refund handbook\nowner: support\n---\n# Refunds\n\nWe refund an unused subscription within 30 days of the charge.\n\n## Eligibility\n\nA refund needs the original order id. Partial months are not refunded.\n\n## How to request one\n\nOpen a ticket from the billing page and attach the invoice.\n","strategy":"HEADING_THEN_TOKEN","limits":{"maxTokens":128},"metadata":{"source":"support-wiki","revision":7},"documentId":"refund-handbook","format":"MARKDOWN"});

Errors

Code	HTTP	Retryable	Description
VALIDATION_ERROR	400	No	The request payload failed schema validation.
UNAUTHORIZED	401	No	Missing or invalid API key.
SKILL_NOT_ACCESSIBLE	403	No	The API key is not granted access to this skill.
SKILL_NOT_AVAILABLE_FOR_PLAN	403	No	This skill is not included in your current plan.
PLAN_ELIGIBILITY_UNAVAILABLE	503	Yes	Plan eligibility could not be decided. Retry after backoff.
SKILL_NOT_FOUND	404	No	The requested skill or version does not exist.
TENANT_RATE_LIMITED	429	Yes	Request rate limit exceeded. Retry after backoff.
QUOTA_EXCEEDED	429	Yes	Plan quota exhausted. Retryable once the quota period resets.
EXECUTION_TIMEOUT	504	Yes	Execution exceeded the allowed time budget.
CHUNK_TEXT_INVALID	400	No	The request payload failed this skill's input validation.
CHUNK_TEXT_TOO_LARGE	429	No	A size, quota, rate, or time limit was exceeded.
CHUNK_LIMITS_INVALID	400	No	The request payload failed this skill's input validation.
CHUNK_LIMIT_EXCEEDED	429	No	A size, quota, rate, or time limit was exceeded.
CHUNK_TOKENIZER_UNSUPPORTED	400	No	The skill does not support the requested variant.
CHUNK_METADATA_INVALID	400	No	The request payload failed this skill's input validation.
CHUNK_OUTPUT_TOO_LARGE	429	No	A size, quota, rate, or time limit was exceeded.

Other limits

Limit	Value
Recommended inline bytes	32 KiB

Limitations

Does not embed anything or call an embedding model
Does not fetch the document — you send the text
Does not store chunks, an index or a cache between calls
Does not summarise, clean or rewrite the text it returns
Does not parse Markdown beyond headings and fenced code blocks
Splitting a plain string when you do not need offsets or headings
Chunking a document you have not loaded yet — there is no fetch here
Producing embeddings, which no skill in this pack does

How to call

POST /v1/execute (requires authentication)

Request bodyjson

{
  "skillKey": "knowledge.chunk_document",
  "input": {
    "text": "---\ntitle: Refund handbook\nowner: support\n---\n# Refunds\n\nWe refund an unused subscription within 30 days of the charge.\n\n## Eligibility\n\nA refund needs the original order id. Partial months are not refunded.\n\n## How to request one\n\nOpen a ticket from the billing page and attach the invoice.\n",
    "strategy": "HEADING_THEN_TOKEN",
    "limits": {
      "maxTokens": 128
    },
    "metadata": {
      "source": "support-wiki",
      "revision": 7
    },
    "documentId": "refund-handbook",
    "format": "MARKDOWN"
  }
}

Examples

A Markdown handbook cut at its headings, with front matter dropped

Inputjson

{
  "text": "---\ntitle: Refund handbook\nowner: support\n---\n# Refunds\n\nWe refund an unused subscription within 30 days of the charge.\n\n## Eligibility\n\nA refund needs the original order id. Partial months are not refunded.\n\n## How to request one\n\nOpen a ticket from the billing page and attach the invoice.\n",
  "strategy": "HEADING_THEN_TOKEN",
  "limits": {
    "maxTokens": 128
  },
  "metadata": {
    "source": "support-wiki",
    "revision": 7
  },
  "documentId": "refund-handbook",
  "format": "MARKDOWN"
}

Expected outputjson

{
  "chunks": [
    {
      "id": "refund-handbook:0eddf89b5fb9bce7:0",
      "ordinal": 0,
      "text": "# Refunds\n\nWe refund an unused subscription within 30 days of the charge.",
      "tokenCount": 17,
      "charStart": 46,
      "charEnd": 119,
      "headingPath": [
        "Refunds"
      ],
      "metadata": {
        "source": "support-wiki",
        "revision": 7
      },
      "flags": []
    },
    {
      "id": "refund-handbook:b379cc9de5982555:1",
      "ordinal": 1,
      "text": "## Eligibility\n\nA refund needs the original order id. Partial months are not refunded.",
      "tokenCount": 18,
      "charStart": 121,
      "charEnd": 207,
      "headingPath": [
        "Refunds",
        "Eligibility"
      ],
      "metadata": {
        "source": "support-wiki",
        "revision": 7
      },
      "flags": []
    },
    {
      "id": "refund-handbook:ebdaa28c67d03c6e:2",
      "ordinal": 2,
      "text": "## How to request one\n\nOpen a ticket from the billing page and attach the invoice.",
      "tokenCount": 18,
      "charStart": 209,
      "charEnd": 291,
      "headingPath": [
        "Refunds",
        "How to request one"
      ],
      "metadata": {
        "source": "support-wiki",
        "revision": 7
      },
      "flags": []
    }
  ],
  "summary": {
    "chunkCount": 3,
    "totalTokens": 53,
    "strategy": "HEADING_THEN_TOKEN",
    "tokenizer": "cl100k_base",
    "tokenizer_version": "jtokkit-1.1.0",
    "truncated": false,
    "frontMatterDetected": true
  }
}

Tip

Each section stays whole because none of them reaches 128 tokens, every chunk carries the heading trail above it, and the YAML block at the top is reported once rather than becoming a chunk.

Fixed token windows with overlap, and ids built from position