Token Counter

Overview

You pass text (or a file_ref) and pick an encoding — cl100k_base or o200k_base; we run the real tokenizer over it and return the exact token count, the encoding and library version we used, and, on request, the token ids themselves (the first 10,000 of them). A token count means nothing without naming the tokenizer, so we always name it.

LLMs cannot count their own tokens reliably. We use the exact tokenizer encodings so you know how much context a string will consume.

When to use it

Scope

Non-goals

• Does not tokenize for non-OpenAI models (Anthropic, Mistral, etc.)
• Does not estimate API costs — token pricing changes frequently
• Does not count words or characters (use text.count_words or text.count_characters)

Properties

Capabilities

Cost & billing

Flat rate per invocation

File handling

File limits

Constraints this skill applies to your input and generated files. The “Limited by” column shows the static source of each cap; none of these can be raised by upgrading your plan.

How we handle your data

Input schema

Output schema

API contract

curl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"text.count_tokens","input":{"text":"You are a helpful assistant.","tokenizer":"cl100k_base"}}'

import { Squillify } from "@squillify/sdk";

const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });

const { output } = await sq.execute("text.count_tokens", {"tokenizer":"cl100k_base","text":"You are a helpful assistant that summarizes technical documents into concise bullet points."});

Errors

Other limits

Limitations

• Does not tokenize for non-OpenAI models (Anthropic, Mistral, etc.)
• Does not estimate API costs — token pricing changes frequently
• Does not count words or characters (use text.count_words or text.count_characters)
• Word or character counting (use text.count_words or text.count_characters)
• Cost estimation that depends on frequently changing model pricing
• Token counting for non-OpenAI model families

How to call

POST /v1/execute (requires authentication)

{
  "skillKey": "text.count_tokens",
  "input": {
    "tokenizer": "cl100k_base",
    "text": "You are a helpful assistant that summarizes technical documents into concise bullet points."
  }
}

Examples

{
  "tokenizer": "cl100k_base",
  "text": "You are a helpful assistant that summarizes technical documents into concise bullet points."
}

{
  "tokenizer_version": "jtokkit-1.1.0",
  "policy": "tokenizer_encoding",
  "count": 14,
  "skill": "text.count_tokens",
  "unit": "token",
  "tokenizer": "cl100k_base",
  "warnings": []
}

{
  "tokenizer": "o200k_base",
  "text": "The quick brown fox jumps over the lazy dog."
}

{
  "tokenizer_version": "jtokkit-1.1.0",
  "policy": "tokenizer_encoding",
  "count": 10,
  "skill": "text.count_tokens",
  "unit": "token",
  "tokenizer": "o200k_base",
  "warnings": []
}

Recommended use cases

• Pre-flight context-window budget checks before sending a prompt
• Validate prompt length against model limits
• Compare token counts across tokenizer encodings

Avoid for

• Word or character counting (use text.count_words or text.count_characters)
• Cost estimation that depends on frequently changing model pricing
• Token counting for non-OpenAI model families

Related skills

Composable with

Safety & privacy

• Runs without calling any external service.
• Your payloads are never stored permanently.
• Telemetry captures metadata only — never your payload contents.
• We never train models on your payloads.
• No human reviews your payloads.

Tags