Retrieved Chunk Deduplicator

Overview

Retrieval from several sources returns the same passage more than once, and paying to embed or prompt with it twice is waste. Send the chunks you retrieved and we hand back the survivors in their original order plus a removal report saying which chunk was a duplicate of which, and how close they were. Three modes: byte-identical text, text identical after normalising case, whitespace, punctuation and Unicode form, or near-duplicate by word-shingle Jaccard overlap against a threshold you choose. You also choose which member of a duplicate group survives — the first, the highest scoring, or the longest. Near-duplicate detection is a threshold judgement rather than a fact, which is why the mode is yours to pick and the similarity comes back with every removal. Reach for data.detect_duplicates instead when you are matching short record fields, where edit distance is the right measure.

Retrieval across several sources returns the same passage repeatedly, and every copy costs tokens in the prompt that follows. Removing them needs a decision about how close counts as the same and which copy to keep, and both belong to you — so both are settings, and every removal comes back with the evidence behind it.

When to use it

Scope

Non-goals

• Does not use edit distance — chunk similarity is a set problem, not a field problem
• Does not rank, re-order or score the survivors
• Does not merge two near-duplicates into one chunk
• Does not chain matches transitively across a whole cluster
• Does not fetch, store or index anything

Properties

Capabilities

Cost & billing

Flat rate per invocation

File handling

How we handle your data

Input schema

Output schema

API contract

curl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"knowledge.deduplicate_chunks","input":{"chunks":[{"id":"c1","text":"Refunds within 30 days.","score":0.9},{"id":"c2","text":"refunds within 30 days","score":0.4}],"mode":"NORMALIZED","keep":"HIGHEST_SCORE"}}'

import { Squillify } from "@squillify/sdk";

const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });

const { output } = await sq.execute("knowledge.deduplicate_chunks", {"chunks":[{"text":"We refund an unused subscription within 30 days.","score":0.91,"id":"a"},{"text":"A refund needs the original order id.","score":0.74,"id":"b"},{"text":"We refund an unused subscription within 30 days.","score":0.55,"id":"c"}],"mode":"EXACT"});

Errors

Other limits

Limitations

• Does not use edit distance — chunk similarity is a set problem, not a field problem
• Does not rank, re-order or score the survivors
• Does not merge two near-duplicates into one chunk
• Does not chain matches transitively across a whole cluster
• Does not fetch, store or index anything
• Matching customer records or short field values, where data.detect_duplicates fits
• Deciding which chunks are most relevant, which knowledge.rank_bm25 answers
• Merging duplicates into a single combined chunk

How to call

POST /v1/execute (requires authentication)

{
  "skillKey": "knowledge.deduplicate_chunks",
  "input": {
    "chunks": [
      {
        "text": "We refund an unused subscription within 30 days.",
        "score": 0.91,
        "id": "a"
      },
      {
        "text": "A refund needs the original order id.",
        "score": 0.74,
        "id": "b"
      },
      {
        "text": "We refund an unused subscription within 30 days.",
        "score": 0.55,
        "id": "c"
      }
    ],
    "mode": "EXACT"
  }
}

Examples

{
  "chunks": [
    {
      "text": "We refund an unused subscription within 30 days.",
      "score": 0.91,
      "id": "a"
    },
    {
      "text": "A refund needs the original order id.",
      "score": 0.74,
      "id": "b"
    },
    {
      "text": "We refund an unused subscription within 30 days.",
      "score": 0.55,
      "id": "c"
    }
  ],
  "mode": "EXACT"
}

{
  "removed": [
    {
      "id": "c",
      "duplicateOfId": "a",
      "mode": "EXACT",
      "reasonCode": "EXACT_MATCH"
    }
  ],
  "summary": {
    "inputCount": 3,
    "keptCount": 2,
    "removedCount": 1,
    "mode": "EXACT",
    "shingleSize": 5,
    "keep": "FIRST",
    "blockBy": "NONE",
    "normalization": {
      "caseFold": true,
      "collapseWhitespace": true,
      "stripPunctuation": false,
      "unicodeForm": "NFKC"
    },
    "comparisons": 0,
    "truncated": false
  },
  "chunks": [
    {
      "id": "a",
      "text": "We refund an unused subscription within 30 days.",
      "score": 0.91
    },
    {
      "id": "b",
      "text": "A refund needs the original order id.",
      "score": 0.74
    }
  ]
}

{
  "chunks": [
    {
      "id": "first",
      "text": "We refund an unused subscription within 30 days of the charge."
    },
    {
      "id": "second",
      "text": "We refund an unused subscription within 30 days of the payment."
    },
    {
      "id": "third",
      "text": "Partial months are never refunded under any circumstances."
    }
  ],
  "threshold": 0.5,
  "mode": "SHINGLE_JACCARD",
  "keep": "FIRST",
  "shingleSize": 3
}

{
  "removed": [
    {
      "id": "second",
      "duplicateOfId": "first",
      "mode": "SHINGLE_JACCARD",
      "similarity": 0.8,
      "reasonCode": "SIMILARITY_ABOVE_THRESHOLD"
    }
  ],
  "summary": {
    "inputCount": 3,
    "keptCount": 2,
    "removedCount": 1,
    "mode": "SHINGLE_JACCARD",
    "threshold": 0.5,
    "shingleSize": 3,
    "keep": "FIRST",
    "blockBy": "NONE",
    "normalization": {
      "caseFold": true,
      "collapseWhitespace": true,
      "stripPunctuation": false,
      "unicodeForm": "NFKC"
    },
    "comparisons": 15,
    "truncated": false
  },
  "chunks": [
    {
      "id": "first",
      "text": "We refund an unused subscription within 30 days of the charge."
    },
    {
      "id": "third",
      "text": "Partial months are never refunded under any circumstances."
    }
  ]
}

Recommended use cases

• Collapse the same passage returned by two retrievers before packing a prompt
• Drop re-formatted copies of a document that differ only in whitespace and case
• Find near-duplicate chunks in a knowledge base before an embedding run

Avoid for

• Matching customer records or short field values, where data.detect_duplicates fits
• Deciding which chunks are most relevant, which knowledge.rank_bm25 answers
• Merging duplicates into a single combined chunk

Related skills

Composable with

Safety & privacy

• Runs without calling any external service.
• Your payloads are never stored permanently.
• Telemetry captures metadata only — never your payload contents.
• We never train models on your payloads.
• No human reviews your payloads.

Tags