Deduplicate Rows

data.deduplicate_rows

ProductionBounded custom (Class B)Deterministicstrictfastv1.0.0

No network

Overview

When you send rows, we remove duplicates by full-row identity or by the key columns you name, keeping the first or last occurrence in stable input order. We never mutate rows in place and always report how many were removed — do not use this for fuzzy or near-duplicate matching.

Exact deduplication should be explainable and repeatable, not guessed.

Reliability

Same rows in, same rows out, and running it twice changes nothing the second time. Comparison is by value and by type: 1, 1.0 and "1" are three different values, and key order inside a row never matters. Two refusals, neither worth a retry with the same payload: key_columns mode without a non-empty keys array — add the columns or drop back to full_row; and a row that does not carry one of your key columns — fill the gap (an explicit null is fine) or switch mode. Sending more than 100,000 rows is refused the same way.

Context footprint

Returns deduplicated rows, counts, and optional removedRows when requested. A successful call costs 1 execution credit.

When to use it

When you need to drop exact duplicate rows or duplicate keys.

When matches may be fuzzy — use data.detect_duplicates first.

Scope

Note

Removes exact duplicate rows with an explicit keep strategy.

Non-goals

Does not fuzzy-match rows
Does not mutate input objects in place

Properties

IdempotentRetry safeStreaming supportedFile input supportedRequires external service

Capabilities

data-transformation

Cost & billing

Flat rate per invocation

MetercallsBillable onsuccessCost per call1 credit

File handling

Accepts file_refProduces file_refMax input97.7 KB

How we handle your data

Sensitive output possible

What this means

This skill may receive sensitive content and may return submitted, transformed, extracted, or reformatted content.

Squillify guarantees

Runs inside Squillify
This skill runs inside Squillify.
Payloads are not sent to an external service by this skill.
Squillify does not permanently store submitted payloads.
Telemetry records metadata only, not submitted content.
Submitted payloads are not used for training.
Submitted payloads are not reviewed by humans.
This skill does not target sensitive data.
The response may contain submitted or transformed content.

Your responsibility

Squillify does not determine whether submitted content is sensitive. Your application is responsible for deciding what data it is allowed to submit. Your application is responsible for securing the response.

Read the Microskill Data Handling Policy

Input schema

Field	Type	Required	Description
rows	array	Yes	Bounded array of 0 through 100000 input row objects keyed by column name, where missing keys count as null.
keys	array	No	Bounded array of identity columns for key_columns mode; omit it for full_row mode.
mode	string	No	How identity is decided; omit it for full_row.
keep	string	No	Which duplicate copy survives; omit it for first.
normalizeStrings	boolean	No	Whether text is trimmed and whitespace-collapsed before compare; omit it for false.
caseSensitive	boolean	No	Whether text comparison is case-sensitive; omit it for true.
includeRemovedRows	boolean	No	Whether dropped rows are returned; omit it for false.

Output schema

Field	Type	Required	Description
rows	array	Yes	Bounded array of 0 through 100000 result row objects keyed by column name after this skill transformed the table.
originalRowCount	integer	Yes	How many rows were sent before anything was dropped.
deduplicatedRowCount	integer	Yes	How many survivor rows came back.
removedCount	integer	Yes	How many rows were dropped.
removedRows	array	No	Bounded array of 0 through 100000 input row objects keyed by column name, where missing keys count as null.

API contract

POST /v1/execute

curl

curl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"data.deduplicate_rows","input":{"rows":[{"a":1},{"a":1}]}}'

Integrations

TypeScript SDK

import { Squillify } from "@squillify/sdk";

const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });

const { output } = await sq.execute("data.deduplicate_rows", {"rows":[{"a":1},{"a":1},{"a":2}]});

Errors

Code	HTTP	Retryable	Description
VALIDATION_ERROR	400	No	The request payload failed schema validation.
UNAUTHORIZED	401	No	Missing or invalid API key.
SKILL_NOT_ACCESSIBLE	403	No	The API key is not granted access to this skill.
SKILL_NOT_AVAILABLE_FOR_PLAN	403	No	This skill is not included in your current plan.
PLAN_ELIGIBILITY_UNAVAILABLE	503	Yes	Plan eligibility could not be decided. Retry after backoff.
SKILL_NOT_FOUND	404	No	The requested skill or version does not exist.
TENANT_RATE_LIMITED	429	Yes	Request rate limit exceeded. Retry after backoff.
QUOTA_EXCEEDED	429	Yes	Plan quota exhausted. Retryable once the quota period resets.
EXECUTION_TIMEOUT	504	Yes	Execution exceeded the allowed time budget.

Other limits

Limit	Value
Recommended inline bytes	32 KiB

Limitations

Does not fuzzy-match rows
Does not mutate input objects in place
Fuzzy or near-duplicate detection

How to call

POST /v1/execute (requires authentication)

Request bodyjson

{
  "skillKey": "data.deduplicate_rows",
  "input": {
    "rows": [
      {
        "a": 1
      },
      {
        "a": 1
      },
      {
        "a": 2
      }
    ]
  }
}

Examples

Remove a full-row duplicate keeping first

Inputjson

{
  "rows": [
    {
      "a": 1
    },
    {
      "a": 1
    },
    {
      "a": 2
    }
  ]
}

Expected outputjson

{
  "originalRowCount": 3,
  "deduplicatedRowCount": 2,
  "removedCount": 1,
  "rows": [
    {
      "a": 1
    },
    {
      "a": 2
    }
  ]
}

Deduplicate by key columns keeping last