Detect Duplicate URLs

Overview

Group supplied URL strings that share the same syntactic identity key after named normalization We never open a connection, resolve DNS, or follow a redirect. The answer is a structured fact or policy result you can store. A successful call costs 1 execution credit.

A pair-wise eye check misses the third copy; this groups the whole supplied list.

When to use it

Scope

Non-goals

• Does not fetch a URL, resolve DNS, or run a browser
• Does not predict ranking, choose a winner, or claim live index status
• Does not compare page bodies or pick a canonical

Properties

Capabilities

Cost & billing

Flat rate per invocation

File handling

How we handle your data

Input schema

Output schema

API contract

curl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"seo.detect_duplicate_urls","input":{"urls":["HTTPS://Example.com/a","https://example.com/a"]}}'

import { Squillify } from "@squillify/sdk";

const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });

const { output } = await sq.execute("seo.detect_duplicate_urls", {"urls":["HTTPS://Example.com/a","https://example.com/a","https://example.com/b"]});

Errors

Other limits

Limitations

• Does not fetch a URL, resolve DNS, or run a browser
• Does not predict ranking, choose a winner, or claim live index status
• Does not compare page bodies or pick a canonical
• Choosing a winner in a duplicate cluster
• Checking live content overlap

How to call

POST /v1/execute (requires authentication)

{
  "skillKey": "seo.detect_duplicate_urls",
  "input": {
    "urls": [
      "HTTPS://Example.com/a",
      "https://example.com/a",
      "https://example.com/b"
    ]
  }
}

Examples

{
  "urls": [
    "HTTPS://Example.com/a",
    "https://example.com/a",
    "https://example.com/b"
  ]
}

{
  "groups": [
    {
      "identityKey": "https://example.com/a",
      "urls": [
        "HTTPS://Example.com/a",
        "https://example.com/a"
      ],
      "count": 2
    }
  ],
  "profile": "seo.safe.url.normalization@1",
  "suppliedCount": 3,
  "includeSingletons": false,
  "uniqueCount": 2,
  "duplicateCount": 1
}

{
  "urls": [
    "https://example.com/a",
    "https://example.com/b"
  ],
  "includeSingletons": true
}

{
  "groups": [
    {
      "identityKey": "https://example.com/a",
      "urls": [
        "https://example.com/a"
      ],
      "count": 1
    },
    {
      "identityKey": "https://example.com/b",
      "urls": [
        "https://example.com/b"
      ],
      "count": 1
    }
  ],
  "profile": "seo.safe.url.normalization@1",
  "suppliedCount": 2,
  "includeSingletons": true,
  "uniqueCount": 2,
  "duplicateCount": 0
}

Recommended use cases

• Group product URLs that differ only by HTTPS vs host case
• Count extra copies after syntactic rewrite

Avoid for

• Choosing a winner in a duplicate cluster
• Checking live content overlap

Related skills

Composable with

Safety & privacy

• Runs without calling any external service.
• Your payloads are never stored permanently.
• Telemetry captures metadata only — never your payload contents.
• We never train models on your payloads.
• No human reviews your payloads.

Tags