Token Chunker

text.chunk_by_tokens

ProductionBounded custom (Class B)Deterministicverifiablefastv1.0.0

No network

Overview

You pass text and a token budget; we split it into chunks whose token count does not exceed maxTokens, with optional token overlap. We return character offsets, token counts, and deterministic chunk IDs. Paragraph preservation is best-effort; long paragraphs are hard-split at token boundaries when enabled. Offsets are UTF-16 Java string indexes. Do not use this for semantic chunking or non-OpenAI tokenizers.

Character-based chunking breaks token budgets unpredictably — you need reproducible token-bounded chunks for retrieval, embeddings, and hand-off.

Reliability

VERIFIABLE and deterministic for identical input and options on this deployment. Tokenizer encodings are version-sensitive — re-run after platform upgrades if you depend on exact chunk boundaries. Input validation (not retryable): missing text; text too long; maxTokens or maxChunks out of range; overlapTokens >= maxTokens; unsupported tokenizer.

Context footprint

Bounded input (2 MB) and maxChunks; character offsets are UTF-16 indexes. A successful call costs 1 execution credit.

When to use it

When you must split text into chunks that fit a token budget with optional overlap.

When you need semantic chunking or tokenizers other than the supported set.

Scope

Note

Splits plain text into token-bounded chunks (cl100k_base/o200k_base via jtokkit, or a simple Unicode word tokenizer) with optional overlap and deterministic IDs.

Non-goals

Does not summarize or embed text
Does not call any language model
Does not parse document formats beyond plain text

Properties

IdempotentRetry safeStreaming supportedFile input supportedRequires external service

Capabilities

data-transformationtext-processing

Cost & billing

Flat rate per invocation

MetercallsBillable onsuccessCost per call1 credit

File handling

Accepts file_refProduces file_refMax input1.9 MB

How we handle your data

Sensitive input, safe output

What this means

This skill may receive sensitive content, but it does not target sensitive data and returns only a structured non-sensitive result.

Squillify guarantees

Runs inside Squillify
This skill runs inside Squillify.
Payloads are not sent to an external service by this skill.
Squillify does not permanently store submitted payloads.
Telemetry records metadata only, not submitted content.
Submitted payloads are not used for training.
Submitted payloads are not reviewed by humans.
This skill does not target sensitive data.
The response does not contain submitted content.

Your responsibility

Squillify does not determine whether submitted content is sensitive. Your application is responsible for deciding what data it is allowed to submit. Your application is responsible for securing the response.

Read the Microskill Data Handling Policy

Input schema

Field	Type	Required	Description
text	string	Yes	Text divided into tokenizer-bounded chunks.
maxTokens	integer	Yes	Maximum tokens in one chunk.
overlapTokens	integer	No	Adjacent-chunk overlap; omit it for zero.
tokenizer	string	No	Tokenizer encoding; omit it for cl100k_base.
preserveParagraphs	boolean	No	Whether paragraph boundaries are preferred; omit it for true.
hardSplitLongSegments	boolean	No	Whether overlong segments are split; omit it for true.
includeText	boolean	No	Whether chunk text is returned; omit it for true.
maxChunks	integer	No	Chunk cap; omit it for one thousand.

Output schema

Field	Type	Required	Description
chunks	array	Yes	Bounded array of chunk receipts in order with index, token bounds, and optional text. The item shape is defined by the nested schema and the array contains from 0 through 10000 items.
chunkCount	integer	Yes	Returned chunk count.
totalTokens	integer	Yes	Original input token count.
tokenizer	string	Yes	Applied tokenizer encoding.

API contract

POST /v1/execute

curl

curl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"text.chunk_by_tokens","input":{"text":"The quick brown fox.","maxTokens":512,"overlapTokens":64}}'

Integrations

TypeScript SDK

import { Squillify } from "@squillify/sdk";

const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });

const { output } = await sq.execute("text.chunk_by_tokens", {"text":"The quick brown fox.","maxTokens":4,"tokenizer":"cl100k_base"});

Errors

Code	HTTP	Retryable	Description
VALIDATION_ERROR	400	No	The request payload failed schema validation.
UNAUTHORIZED	401	No	Missing or invalid API key.
SKILL_NOT_ACCESSIBLE	403	No	The API key is not granted access to this skill.
SKILL_NOT_AVAILABLE_FOR_PLAN	403	No	This skill is not included in your current plan.
PLAN_ELIGIBILITY_UNAVAILABLE	503	Yes	Plan eligibility could not be decided. Retry after backoff.
SKILL_NOT_FOUND	404	No	The requested skill or version does not exist.
TENANT_RATE_LIMITED	429	Yes	Request rate limit exceeded. Retry after backoff.
QUOTA_EXCEEDED	429	Yes	Plan quota exhausted. Retryable once the quota period resets.
EXECUTION_TIMEOUT	504	Yes	Execution exceeded the allowed time budget.

Other limits

Limit	Value
Recommended inline bytes	32 KiB

Limitations

Does not summarize or embed text
Does not call any language model
Does not parse document formats beyond plain text
Semantic/section-aware chunking that needs understanding
Chunking by characters or words (use character/word skills)

How to call

POST /v1/execute (requires authentication)

Request bodyjson

{
  "skillKey": "text.chunk_by_tokens",
  "input": {
    "text": "The quick brown fox.",
    "maxTokens": 4,
    "tokenizer": "cl100k_base"
  }
}

Examples

Chunk a short sentence

Inputjson

{
  "text": "The quick brown fox.",
  "maxTokens": 4,
  "tokenizer": "cl100k_base"
}

Expected outputjson

{
  "totalTokens": 5,
  "tokenizer": "cl100k_base",
  "chunks": [
    {
      "tokenCount": 4,
      "id": "63a8ef50989ffb3e",
      "text": "The quick brown fox",
      "endChar": 19,
      "overlapWithPreviousTokens": 0,
      "index": 0,
      "startChar": 0
    },
    {
      "tokenCount": 1,
      "id": "68feee9f6c1e5ea9",
      "text": ".",
      "endChar": 20,
      "overlapWithPreviousTokens": 0,
      "index": 1,
      "startChar": 19
    }
  ],
  "chunkCount": 2
}

Single chunk within budget