BM25 Document Ranker

Overview

Give us a query and the documents you want ranked, and we score each one with Okapi BM25 and return them best first. The formula and the analyzer are both published, so you can reproduce a score by hand: the non-negative inverse-document-frequency variant, and term splitting you control through a closed set of patterns rather than a regular expression. Corpus statistics come from the documents in your request and nowhere else, so a score is relative to the set you sent rather than an absolute measure of relevance. Nothing here is language-model tokenization and no tokenizer is involved: BM25 counts words, and subword units would give scores no BM25 reference agrees with. Documents matching nothing are counted rather than padding the ranking.

Lexical ranking is still the cheapest way to tell a relevant passage from a plausible one, and it only earns trust when the arithmetic is inspectable. Both the formula and the analyzer are published here, and the per-term breakdown is available on request, so a ranking you disagree with can be checked rather than argued with.

When to use it

Scope

Non-goals

• Does not use the platform tokenizer — BM25 counts words, not model tokens
• Does not hold an index or remember a document between calls
• Does not compile a regular expression you supply
• Does not ship a built-in stopword list for any language
• Does not compute embeddings or semantic similarity

Properties

Capabilities

Cost & billing

Flat rate per invocation

File handling

How we handle your data

Input schema

Output schema

API contract

curl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"knowledge.rank_bm25","input":{"query":"refund policy","documents":[{"id":"d1","text":"Our refund policy covers 30 days."},{"id":"d2","text":"Shipping policy and delivery times."}],"k1":1.2,"b":0.75,"limit":10}}'

import { Squillify } from "@squillify/sdk";

const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });

const { output } = await sq.execute("knowledge.rank_bm25", {"query":"refund policy","documents":[{"id":"doc-refunds","text":"Our refund policy: we refund an unused subscription within 30 days."},{"id":"doc-shipping","text":"Shipping takes three days and the policy covers lost parcels."},{"id":"doc-privacy","text":"We store personal data only as long as the law requires."}]});

Errors

Other limits

Limitations

• Does not use the platform tokenizer — BM25 counts words, not model tokens
• Does not hold an index or remember a document between calls
• Does not compile a regular expression you supply
• Does not ship a built-in stopword list for any language
• Does not compute embeddings or semantic similarity
• Searching a corpus you have not sent — there is no index here
• Semantic matching where wording differs but meaning does not
• Comparing scores between two different requests, which BM25 does not support

How to call

POST /v1/execute (requires authentication)

{
  "skillKey": "knowledge.rank_bm25",
  "input": {
    "query": "refund policy",
    "documents": [
      {
        "id": "doc-refunds",
        "text": "Our refund policy: we refund an unused subscription within 30 days."
      },
      {
        "id": "doc-shipping",
        "text": "Shipping takes three days and the policy covers lost parcels."
      },
      {
        "id": "doc-privacy",
        "text": "We store personal data only as long as the law requires."
      }
    ]
  }
}

Examples

{
  "query": "refund policy",
  "documents": [
    {
      "id": "doc-refunds",
      "text": "Our refund policy: we refund an unused subscription within 30 days."
    },
    {
      "id": "doc-shipping",
      "text": "Shipping takes three days and the policy covers lost parcels."
    },
    {
      "id": "doc-privacy",
      "text": "We store personal data only as long as the law requires."
    }
  ]
}

{
  "summary": {
    "documentCount": 3,
    "queryTerms": [
      "refund",
      "policy"
    ],
    "droppedQueryTerms": [],
    "averageDocumentLength": 10.666667,
    "matchedCount": 2,
    "unmatchedCount": 1,
    "k1": 1.2,
    "b": 0.75,
    "titleBoost": 0,
    "analyzer": {
      "caseFold": true,
      "unicodeForm": "NFKC",
      "tokenPattern": "UNICODE_WORD",
      "stopwordCount": 0,
      "minTermLength": 1
    },
    "truncated": false
  },
  "ranked": [
    {
      "rank": 1,
      "id": "doc-refunds",
      "score": 1.800961,
      "matchedTerms": [
        "refund",
        "policy"
      ]
    },
    {
      "rank": 2,
      "id": "doc-shipping",
      "score": 0.482336,
      "matchedTerms": [
        "policy"
      ]
    }
  ]
}

{
  "analyzer": {
    "stopwords": [
      "the",
      "of",
      "a"
    ],
    "minTermLength": 2
  },
  "query": "the refund of a subscription",
  "documents": [
    {
      "text": "Refund requests are reviewed within two days.",
      "title": "Refund handling",
      "id": "a"
    },
    {
      "id": "b",
      "text": "A subscription renews monthly until it is cancelled by the customer."
    }
  ],
  "includeTermScores": true,
  "b": 0,
  "titleBoost": 2
}

{
  "summary": {
    "documentCount": 2,
    "queryTerms": [
      "refund",
      "subscription"
    ],
    "droppedQueryTerms": [
      "the",
      "of",
      "a"
    ],
    "averageDocumentLength": 8,
    "matchedCount": 2,
    "unmatchedCount": 0,
    "k1": 1.2,
    "b": 0,
    "titleBoost": 2,
    "analyzer": {
      "caseFold": true,
      "unicodeForm": "NFKC",
      "tokenPattern": "UNICODE_WORD",
      "stopwordCount": 3,
      "minTermLength": 2
    },
    "truncated": false
  },
  "ranked": [
    {
      "rank": 1,
      "id": "a",
      "score": 1.089231,
      "matchedTerms": [
        "refund"
      ],
      "termScores": [
        {
          "term": "refund",
          "idf": 0.693147,
          "termFrequency": 3,
          "contribution": 1.089231
        }
      ]
    },
    {
      "rank": 2,
      "id": "b",
      "score": 0.693147,
      "matchedTerms": [
        "subscription"
      ],
      "termScores": [
        {
          "term": "subscription",
          "idf": 0.693147,
          "termFrequency": 1,
          "contribution": 0.693147
        }
      ]
    }
  ]
}

Recommended use cases

• Rerank a shortlist a vector search returned, using lexical evidence
• Rank a small document set without standing up a search engine
• Explain why one document outranks another, with per-term contributions

Avoid for

• Searching a corpus you have not sent — there is no index here
• Semantic matching where wording differs but meaning does not
• Comparing scores between two different requests, which BM25 does not support

Related skills

Composable with

Safety & privacy

• Runs without calling any external service.
• Your payloads are never stored permanently.
• Telemetry captures metadata only — never your payload contents.
• We never train models on your payloads.
• No human reviews your payloads.

Tags