Detect Data Drift

data.detect_drift

ProductionBounded custom (Class B)Deterministicstrictfastv1.0.0

No network

Overview

You send two column profiles of the same published shape — a current one and a baseline — and we report, per column, how far each statistic moved and how serious that is on a five-grade scale. Numeric columns get up to eight metrics, text and date columns fewer, because a metric a column's values cannot answer is worse than a missing one. A column that appears or disappears is reported as a structural change with no metrics rather than scored as total drift, and a metric we could not compute comes back as null and is named, because could-not-tell and did-not-move are different facts. We never read the sample values a profile may carry, and we never return them.

A profile you never compare against anything is a number you filed away; the value is in the delta and in knowing which delta matters.

Reliability

The same two profiles always give the same grades, on any machine and under any locale, time zone or default charset. Two published facts about the baseline are worth knowing before you act on a grade. First, data.profile_dataset profiles a sample of at most 10,000 rows by default, so part of any movement between two differently sampled profiles is sampling rather than data. Second, its type inference is heuristic and it says so, which means a TYPE_CHANGED finding can be an artefact of inference on a borderline column rather than a real schema change — it is graded CRITICAL because the cost of ignoring a real one is high, not because we are certain. Band edges belong to the higher grade, so a value sitting exactly on the critical edge is CRITICAL. Ten refusals, none retryable unchanged: DRIFT_PROFILE_INVALID for a profile that is not the published shape; DRIFT_PROFILE_VERSION_MISSING and DRIFT_PROFILE_VERSION_UNSUPPORTED, both refusals rather than a best-effort comparison, because grading against a baseline whose semantics are unknown invents a number; DRIFT_COLUMN_LIMIT_EXCEEDED above 5000 columns on either side; DRIFT_COLUMN_NAME_DUPLICATE when one profile names a column twice; DRIFT_COLUMN_TYPE_UNSUPPORTED for a type outside the published eight; DRIFT_METRIC_UNSUPPORTED for an includeMetrics list that is empty, longer than eight or holds an unknown name; DRIFT_THRESHOLDS_INVALID for a band that does not strictly ascend or falls outside 0 to 1000; DRIFT_WORK_REFUSED when the column union times the metrics per column passes 20000 comparisons, reported with both numbers; and DRIFT_OUTPUT_TOO_LARGE when the report would not fit the inline response ceiling.

Context footprint

Returns one entry per column with its metrics and grade, plus counts — never a row, a value or a profile example. A successful call costs 1 execution credit.

When to use it

When you already profile a table on a schedule and want the differences graded rather than eyeballed.

When you have the rows rather than the profiles — profile them first.

Scope

Note

Grades the difference between two column profiles of the same shape.

Non-goals

Does not profile a dataset — data.profile_dataset does that
Does not read or return the sample values a profile may carry
Does not say why a column moved
Does not decide whether a model should be retrained

Properties

IdempotentRetry safeStreaming supportedFile input supportedRequires external service

Capabilities

analysis

Cost & billing

Flat rate per invocation

MetercallsBillable onsuccessCost per call1 credit

File handling

Accepts file_refProduces file_refMax input1 MB

How we handle your data

Structured input

What this means

This skill only accepts structured technical values such as numbers, booleans, enums, timestamps, or identifiers. It does not process free-form text, files, or documents.

Squillify guarantees

Runs inside Squillify
This skill runs inside Squillify.
Payloads are not sent to an external service by this skill.
Squillify does not permanently store submitted payloads.
Telemetry records metadata only, not submitted content.
Submitted payloads are not used for training.
Submitted payloads are not reviewed by humans.
This skill does not target sensitive data.
The response does not contain submitted content.

Your responsibility

Squillify does not determine whether submitted content is sensitive. Your application is responsible for deciding what data it is allowed to submit. Your application is responsible for securing the response.

Read the Microskill Data Handling Policy

Input schema

Field	Type	Required	Description
currentProfile	object	Yes	Closed current column profile of the published data.column_profile/1 shape.
baselineProfile	object	Yes	Closed baseline column profile of the same published shape and limits.
thresholds	object	No	Purpose-bound metric-band overrides keyed by metric name; extra keys are metric names and values are closed band objects. Omit it to keep the published bands.
includeMetrics	array	No	Bounded array of 1 through 8 metric names narrowing what is computed; omit it for every metric the column type supports.

Output schema

Field	Type	Required	Description
drift	array	Yes	Bounded array of 0 through 10000 closed column drift entries ordered by severity then name.
severity	string	Yes	Worst grade across every column.
summary	object	Yes	Closed comparison counts including structural changes and truncation.
_meta	object	Yes	Closed provenance naming the profile shape and which bands produced the grades.

API contract

POST /v1/execute

curl

curl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"data.detect_drift","input":{"currentProfile":{"profileSchemaVersion":"data.column_profile/1","rowCount":1000,"columns":[{"name":"amount","inferredType":"decimal","nullCount":300,"nullRatio":0.3,"distinctCount":700}]},"baselineProfile":{"profileSchemaVersion":"data.column_profile/1","rowCount":1000,"columns":[{"name":"amount","inferredType":"decimal","nullCount":10,"nullRatio":0.01,"distinctCount":990}]}}}'

Integrations

TypeScript SDK

import { Squillify } from "@squillify/sdk";

const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });

const { output } = await sq.execute("data.detect_drift", {"currentProfile":{"rowCount":1000,"profileSchemaVersion":"data.column_profile/1","columns":[{"inferredType":"decimal","name":"amount","nullCount":300,"nullRatio":0.3,"distinctCount":700}]},"includeMetrics":["nullRatioDelta"],"baselineProfile":{"rowCount":1000,"profileSchemaVersion":"data.column_profile/1","columns":[{"inferredType":"decimal","name":"amount","nullCount":10,"nullRatio":0.01,"distinctCount":990}]}});

Errors

Code	HTTP	Retryable	Description
VALIDATION_ERROR	400	No	The request payload failed schema validation.
UNAUTHORIZED	401	No	Missing or invalid API key.
SKILL_NOT_ACCESSIBLE	403	No	The API key is not granted access to this skill.
SKILL_NOT_AVAILABLE_FOR_PLAN	403	No	This skill is not included in your current plan.
PLAN_ELIGIBILITY_UNAVAILABLE	503	Yes	Plan eligibility could not be decided. Retry after backoff.
SKILL_NOT_FOUND	404	No	The requested skill or version does not exist.
TENANT_RATE_LIMITED	429	Yes	Request rate limit exceeded. Retry after backoff.
QUOTA_EXCEEDED	429	Yes	Plan quota exhausted. Retryable once the quota period resets.
EXECUTION_TIMEOUT	504	Yes	Execution exceeded the allowed time budget.
DRIFT_PROFILE_INVALID	400	No	The request payload failed this skill's input validation.
DRIFT_PROFILE_VERSION_MISSING	400	No	The request payload failed this skill's input validation.
DRIFT_PROFILE_VERSION_UNSUPPORTED	400	No	The skill does not support the requested variant.
DRIFT_COLUMN_LIMIT_EXCEEDED	429	No	A size, quota, rate, or time limit was exceeded.
DRIFT_COLUMN_NAME_DUPLICATE	400	No	The request payload failed this skill's input validation.
DRIFT_COLUMN_TYPE_UNSUPPORTED	400	No	The skill does not support the requested variant.
DRIFT_METRIC_UNSUPPORTED	400	No	The skill does not support the requested variant.
DRIFT_THRESHOLDS_INVALID	400	No	The request payload failed this skill's input validation.
DRIFT_WORK_REFUSED	429	No	A size, quota, rate, or time limit was exceeded.
DRIFT_OUTPUT_TOO_LARGE	429	No	A size, quota, rate, or time limit was exceeded.

Other limits

Limit	Value
Recommended inline bytes	32 KiB

Limitations

Does not profile a dataset — data.profile_dataset does that
Does not read or return the sample values a profile may carry
Does not say why a column moved
Does not decide whether a model should be retrained
Comparing profiles produced by different profilers
Treating a grade as a statistical test of distribution equality

How to call

POST /v1/execute (requires authentication)

Request bodyjson

{
  "skillKey": "data.detect_drift",
  "input": {
    "currentProfile": {
      "rowCount": 1000,
      "profileSchemaVersion": "data.column_profile/1",
      "columns": [
        {
          "inferredType": "decimal",
          "name": "amount",
          "nullCount": 300,
          "nullRatio": 0.3,
          "distinctCount": 700
        }
      ]
    },
    "includeMetrics": [
      "nullRatioDelta"
    ],
    "baselineProfile": {
      "rowCount": 1000,
      "profileSchemaVersion": "data.column_profile/1",
      "columns": [
        {
          "inferredType": "decimal",
          "name": "amount",
          "nullCount": 10,
          "nullRatio": 0.01,
          "distinctCount": 990
        }
      ]
    }
  }
}

Examples

A numeric column whose nulls went up

Inputjson

{
  "currentProfile": {
    "rowCount": 1000,
    "profileSchemaVersion": "data.column_profile/1",
    "columns": [
      {
        "inferredType": "decimal",
        "name": "amount",
        "nullCount": 300,
        "nullRatio": 0.3,
        "distinctCount": 700
      }
    ]
  },
  "includeMetrics": [
    "nullRatioDelta"
  ],
  "baselineProfile": {
    "rowCount": 1000,
    "profileSchemaVersion": "data.column_profile/1",
    "columns": [
      {
        "inferredType": "decimal",
        "name": "amount",
        "nullCount": 10,
        "nullRatio": 0.01,
        "distinctCount": 990
      }
    ]
  }
}

Expected outputjson

{
  "_meta": {
    "profileSchemaVersion": "data.column_profile/1",
    "thresholdProfile": "published"
  },
  "severity": "HIGH",
  "summary": {
    "columnsCompared": 1,
    "typeChanges": 0,
    "truncated": false,
    "columnsAdded": 0,
    "columnsRemoved": 0,
    "bySeverity": {
      "CRITICAL": 0,
      "HIGH": 1,
      "LOW": 0,
      "MEDIUM": 0,
      "NONE": 0
    }
  },
  "drift": [
    {
      "metrics": {
        "nullRatioDelta": 0.29
      },
      "kind": "COMPARED",
      "unavailableMetrics": [],
      "severity": "HIGH",
      "column": "amount",
      "columnType": "decimal"
    }
  ]
}

The same movement under caller-defined bands