Token Truncator

Overview

Truncates text so its token count does not exceed maxTokens, using a specified tokenizer encoding (cl100k_base, o200k_base). The returned text is decoded from a token slice whose boundary is proven by roundtrip (encode of the returned text equals the returned slice), so it is never mid-character and never over budget. direction is KEEP_START (keep the beginning) or KEEP_END (keep the end). Because a boundary token may be dropped to stay on a valid boundary, resultTokenCount can be < maxTokens. Accepts file input via file_ref — no need to extract file content first.

Naive token slicing can split a multi-byte character — we prove the boundary by roundtrip so your truncated text never lands mid-character and never exceeds the budget.

When to use it

Scope

Non-goals

• Does not summarize or rewrite text — it only cuts on a token boundary
• Does not tokenize for non-OpenAI models
• Does not chunk text (use text.chunk_by_tokens) or count tokens (use text.count_tokens)

Properties

Capabilities

Cost & billing

Flat rate per invocation

File handling

File limits

Constraints this skill applies to your input and generated files. The “Limited by” column shows the static source of each cap; none of these can be raised by upgrading your plan.

How we handle your data

Input schema

Output schema

API contract

curl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"text.truncate_to_tokens","input":{"text":"Hello world","tokenizer":"o200k_base","maxTokens":4000,"direction":"KEEP_END"}}'

import { Squillify } from "@squillify/sdk";

const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });

const { output } = await sq.execute("text.truncate_to_tokens", {"text":"The quick brown fox jumps over the lazy dog.","maxTokens":4,"tokenizer":"cl100k_base","direction":"KEEP_START"});

Errors

Other limits

Limitations

• Does not summarize or rewrite text — it only cuts on a token boundary
• Does not tokenize for non-OpenAI models
• Does not chunk text (use text.chunk_by_tokens) or count tokens (use text.count_tokens)
• Semantic truncation that needs understanding
• Truncation by characters or words (use character/word skills)

How to call

POST /v1/execute (requires authentication)

{
  "skillKey": "text.truncate_to_tokens",
  "input": {
    "text": "The quick brown fox jumps over the lazy dog.",
    "maxTokens": 4,
    "tokenizer": "cl100k_base",
    "direction": "KEEP_START"
  }
}

Examples

{
  "text": "The quick brown fox jumps over the lazy dog.",
  "maxTokens": 4,
  "tokenizer": "cl100k_base",
  "direction": "KEEP_START"
}

{
  "removedTokenCount": 6,
  "tokenizer": "cl100k_base",
  "boundaryAdjusted": false,
  "boundaryTokensDropped": 0,
  "warnings": [],
  "requestedMaxTokens": 4,
  "originalTokenCount": 10,
  "skill": "text.truncate_to_tokens",
  "tokenizer_version": "jtokkit-1.1.0",
  "text": "The quick brown fox",
  "outputStorage": "INLINE",
  "direction": "KEEP_START",
  "resultTokenCount": 4,
  "truncated": true
}

{
  "text": "Hello world",
  "maxTokens": 100,
  "tokenizer": "cl100k_base",
  "direction": "KEEP_START"
}

{
  "removedTokenCount": 0,
  "tokenizer": "cl100k_base",
  "boundaryAdjusted": false,
  "boundaryTokensDropped": 0,
  "warnings": [],
  "requestedMaxTokens": 100,
  "originalTokenCount": 2,
  "skill": "text.truncate_to_tokens",
  "tokenizer_version": "jtokkit-1.1.0",
  "text": "Hello world",
  "outputStorage": "INLINE",
  "direction": "KEEP_START",
  "resultTokenCount": 2,
  "truncated": false
}

Recommended use cases

• Fit a document into a model context window before a call
• Cap retrieved context to a token budget
• Keep the most recent text within a token limit (KEEP_END)

Avoid for

• Semantic truncation that needs understanding
• Truncation by characters or words (use character/word skills)

Related skills

Composable with

Safety & privacy

• Runs without calling any external service.
• Your payloads are never stored permanently.
• Telemetry captures metadata only — never your payload contents.
• We never train models on your payloads.
• No human reviews your payloads.

Tags