Line Deduplicator

Overview

Removes duplicate lines from text using exact or normalized comparison. Supports case-insensitive matching, Unicode normalization, keep-first or keep-last policy, and configurable line endings. Reports removed lines with details. Accepts file input via file_ref — no need to extract file content first.

Agents frequently produce or collect lists with duplicates. We give you deterministic, policy-driven deduplication across encodings and normalization forms.

When to use it

Scope

Non-goals

• Does not deduplicate tokens within a single line
• Does not merge similar or fuzzy-matched lines
• Does not perform semantic deduplication

Properties

Capabilities

Cost & billing

Flat rate per invocation

File handling

File limits

Constraints this skill applies to your input and generated files. The “Limited by” column shows the static source of each cap; none of these can be raised by upgrading your plan.

How we handle your data

Input schema

Output schema

API contract

curl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"text.deduplicate_lines","input":{"text":"apple\nbanana\napple\ncherry","mode":"exact","keep":"first"}}'

import { Squillify } from "@squillify/sdk";

const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });

const { output } = await sq.execute("text.deduplicate_lines", {"text":"apple\nbanana\napple\ncherry"});

Errors

Other limits

Limitations

• Does not deduplicate tokens within a single line
• Does not merge similar or fuzzy-matched lines
• Does not perform semantic deduplication
• Fuzzy or semantic deduplication
• Within-line token deduplication
• Merging lines that differ only by formatting you have not normalized

How to call

POST /v1/execute (requires authentication)

{
  "skillKey": "text.deduplicate_lines",
  "input": {
    "text": "apple\nbanana\napple\ncherry"
  }
}

Examples

{
  "text": "apple\nbanana\napple\ncherry"
}

{
  "text": "apple\nbanana\ncherry",
  "removedCount": 1,
  "keep": "first",
  "removedDetails": [
    {
      "line": 3,
      "duplicateOf": 1,
      "content": "apple"
    }
  ],
  "keptLines": 3,
  "totalLines": 4,
  "mode": "exact"
}

{
  "caseSensitive": false,
  "text": "Apple\napple\nBANANA"
}

{
  "text": "Apple\nBANANA",
  "removedCount": 1,
  "keep": "first",
  "removedDetails": [
    {
      "line": 2,
      "duplicateOf": 1,
      "content": "apple"
    }
  ],
  "keptLines": 2,
  "totalLines": 3,
  "mode": "exact"
}

Recommended use cases

• Clean up log files with repeated entries
• Deduplicate exported lists or CSV-like text
• Remove redundant lines from generated or scraped content

Avoid for

• Fuzzy or semantic deduplication
• Within-line token deduplication
• Merging lines that differ only by formatting you have not normalized

Related skills

Composable with

Safety & privacy

• Runs without calling any external service.
• Your payloads are never stored permanently.
• Telemetry captures metadata only — never your payload contents.
• We never train models on your payloads.
• No human reviews your payloads.

Tags