Detect Duplicates

data.detect_duplicates

ProductionHeuristic (Class C)Deterministicbest-effortfastv1.0.0

No network

Overview

When you send rows and weighted field rules, we surface probable duplicate clusters with scores and reasons — we never delete or merge rows. Matching supports exact, casefold, normalized, levenshtein, jaro_winkler, and soundex methods with optional blocking fields. Scores are rule-weighted similarity, not calibrated probabilities — do not auto-delete based on results alone.

Near-duplicate review needs transparent scores, not silent merges.

Reliability

The same rows and rules always give the same clusters, but the scores are weighted similarity, not calibrated probabilities — treat them as a ranking for review, never as proof. Clustering is transitive: if A matches B and B matches C, all three land in one group even though A and C were never compared, and the score you see is the best pair inside the group. levenshtein gives up on values longer than 2,000 characters and falls back to a plain equality check. When truncated comes back true we stopped at maxPairs — raise it, or add blockingFields, and call again. Three refusals, none retryable unchanged: rules missing or empty; a rule naming a method outside the six we support, or a negative weight; and every weight set to 0, which leaves nothing to score. Fix the rules and call again.

Context footprint

Returns clusters with rowIndexes, score, reasons, pair count, and truncated flag. A successful call costs 1 execution credit.

When to use it

When you want candidate duplicate clusters before a human or policy decision.

When you need automatic deduplication — use data.deduplicate_rows for exact dupes.

Scope

Note

Finds probable duplicate row clusters for human review.

Non-goals

Does not delete or merge rows
Does not guarantee precision or recall

Properties

IdempotentRetry safeStreaming supportedFile input supportedRequires external service

Capabilities

analysisdata-transformation

Cost & billing

Flat rate per invocation

MetercallsBillable onsuccessCost per call1 credit

File handling

Accepts file_refProduces file_refMax input48.8 KB

How we handle your data

Sensitive input, safe output

What this means

This skill may receive sensitive content, but it does not target sensitive data and returns only a structured non-sensitive result.

Squillify guarantees

Runs inside Squillify
This skill runs inside Squillify.
Payloads are not sent to an external service by this skill.
Squillify does not permanently store submitted payloads.
Telemetry records metadata only, not submitted content.
Submitted payloads are not used for training.
Submitted payloads are not reviewed by humans.
This skill does not target sensitive data.
The response does not contain submitted content.

Your responsibility

Squillify does not determine whether submitted content is sensitive. Your application is responsible for deciding what data it is allowed to submit. Your application is responsible for securing the response.

Read the Microskill Data Handling Policy

Input schema

Field	Type	Required	Description
rows	array	Yes	Bounded array of 0 through 50000 input row objects keyed by column name, where missing keys count as null.
rules	array	Yes	Bounded non-empty array of 1 through 64 closed matching rules whose weights must sum to a positive value.
overallThreshold	number	No	Weighted score two rows must reach to cluster; omit it for 0.9.
blockingFields	array	No	Bounded array of 0 through 32 column names that must match before a pair is scored; omit it to compare every pair.
maxPairs	integer	No	How many pairs are compared before truncation; omit it for 10000.

Output schema

Field	Type	Required	Description
ok	boolean	Yes	Always true on a successful call.
clusters	array	Yes	Bounded array of 0 through 50000 closed clusters ordered by first row index.
candidatePairsEvaluated	integer	Yes	How many row pairs were actually compared after blocking.
truncated	boolean	Yes	True when comparison stopped at maxPairs.

API contract

POST /v1/execute

curl

curl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"data.detect_duplicates","input":{"rows":[{"name":"Acme"},{"name":"Acme"}],"rules":[{"field":"name","method":"exact"}]}}'

Integrations

TypeScript SDK

import { Squillify } from "@squillify/sdk";

const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });

const { output } = await sq.execute("data.detect_duplicates", {"rules":[{"field":"name","method":"exact"}],"rows":[{"name":"Acme"},{"name":"Acme"}]});

Errors

Code	HTTP	Retryable	Description
VALIDATION_ERROR	400	No	The request payload failed schema validation.
UNAUTHORIZED	401	No	Missing or invalid API key.
SKILL_NOT_ACCESSIBLE	403	No	The API key is not granted access to this skill.
SKILL_NOT_AVAILABLE_FOR_PLAN	403	No	This skill is not included in your current plan.
PLAN_ELIGIBILITY_UNAVAILABLE	503	Yes	Plan eligibility could not be decided. Retry after backoff.
SKILL_NOT_FOUND	404	No	The requested skill or version does not exist.
TENANT_RATE_LIMITED	429	Yes	Request rate limit exceeded. Retry after backoff.
QUOTA_EXCEEDED	429	Yes	Plan quota exhausted. Retryable once the quota period resets.
EXECUTION_TIMEOUT	504	Yes	Execution exceeded the allowed time budget.

Other limits

Limit	Value
Recommended inline bytes	32 KiB

Limitations

Does not delete or merge rows
Does not guarantee precision or recall
Automatic deletion or merging

How to call

POST /v1/execute (requires authentication)

Request bodyjson

{
  "skillKey": "data.detect_duplicates",
  "input": {
    "rules": [
      {
        "field": "name",
        "method": "exact"
      }
    ],
    "rows": [
      {
        "name": "Acme"
      },
      {
        "name": "Acme"
      }
    ]
  }
}

Examples

Exact name duplicates

Inputjson

{
  "rules": [
    {
      "field": "name",
      "method": "exact"
    }
  ],
  "rows": [
    {
      "name": "Acme"
    },
    {
      "name": "Acme"
    }
  ]
}

Expected outputjson

{
  "ok": true,
  "candidatePairsEvaluated": 1,
  "truncated": false,
  "clusters": [
    {
      "score": 1,
      "reasons": [
        "name (exact)"
      ],
      "rowIndexes": [
        0,
        1
      ]
    }
  ]
}

Fuzzy name match with threshold