Retrieval Metrics

knowledge.retrieval_metrics

ProductionBounded custom (Class B)Deterministicstrictmoderatev1.0.0

No network

Overview

You send queries with ranked ids and non-negative graded judgments, plus cutoffs and the metrics to compute. Unjudged documents are non-relevant. Precision at k uses k as the denominator even when the ranking is shorter. A query with no relevant judgments yields recall, MRR, MAP and nDCG of 0 so the macro average stays defined. This does not retrieve documents or run a significance test.

A ranking without a published metric is an anecdote; these formulas are the ones IR evaluations already share.

Reliability

Unjudged documents are non-relevant. Precision uses the cutoff as denominator. RETRIEVAL_METRICS_INVALID — trigger: empty queries, unknown metric, or a negative grade; retryable: no; next: send published enums and non-negative grades. RETRIEVAL_METRICS_DUPLICATE_ID — trigger: a repeated query, ranked or judgment id; retryable: no; next: deduplicate. RETRIEVAL_METRICS_LIMIT_EXCEEDED — trigger: more than 1000 queries, 16 cutoffs, cutoff 1001, or 100000 ranked ids; retryable: no; next: split the request.

Context footprint

Ids, counts and rounded metrics only. A successful call costs 2 execution credits.

When to use it

When ranked ids and judgments already exist and standard IR metrics must be computed.

When documents still need retrieving, or when a significance test is the next step.

Powered by Squillify's own IR metric arithmetic (binary and graded nDCG with log2 discounts). No search engine is involved.

Scope

Note

Computes PRECISION, RECALL, MRR, MAP and nDCG for ranked ids against graded judgments.

Non-goals

Does not retrieve documents
Does not run a significance test
Does not pool judgments

Properties

IdempotentRetry safeStreaming supportedFile input supportedRequires external service

Capabilities

analysissearch

Cost & billing

Flat rate per invocation

MetercallsBillable onsuccessCost per call2 credits

File handling

Accepts file_refProduces file_refMax input2 MB

How we handle your data

Sensitive processing

What this means

This skill is designed to detect, mask, redact, validate, or otherwise process sensitive data.

Squillify guarantees

Runs inside Squillify
This skill runs inside Squillify.
Payloads are not sent to an external service by this skill.
Squillify does not permanently store submitted payloads.
Telemetry records metadata only, not submitted content.
Submitted payloads are not used for training.
Submitted payloads are not reviewed by humans.
Targets sensitive data
The response does not contain submitted content.

Your responsibility

Squillify does not determine whether submitted content is sensitive. Your application is responsible for deciding what data it is allowed to submit. Your application is responsible for securing the response.

Read the Microskill Data Handling Policy

Input schema

Field	Type	Required	Description
queries	array	Yes	From 1 through 1001 closed queries; runtime accepts at most 1000.
cutoffs	array	Yes	Distinct cutoffs, 1 through 17 integers; runtime accepts at most 16 values of at most 1000.
metrics	array	Yes	Requested metrics, 1 through 5 unique values from PRECISION, RECALL, MRR, MAP, NDCG.

Output schema

Field	Type	Required	Description
queries	array	Yes	Per-query metric rows in request order, 1 through 1000 objects.
macro	array	Yes	Macro averages of requested metrics at each cutoff, 1 through 16 closed {cutoff, optional precision/recall/mrr/map/ndcg} objects in cutoff order.
unjudgedPolicy	string	Yes	Always NON_RELEVANT in v1.
formulaVersion	string	Yes	Pinned retrieval-metrics formula identifier.

API contract

POST /v1/execute

curl

curl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"knowledge.retrieval_metrics","input":{"queries":[{"id":"q1","ranked":["d1","d2"],"judgments":[{"id":"d1","grade":1}]}],"cutoffs":[1],"metrics":["PRECISION"]}}'

Integrations

TypeScript SDK

import { Squillify } from "@squillify/sdk";

const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });

const { output } = await sq.execute("knowledge.retrieval_metrics", {"queries":[{"judgments":[{"id":"d1","grade":1},{"id":"d3","grade":1}],"id":"q1","ranked":["d1","d2","d3"]}],"cutoffs":[1,3],"metrics":["PRECISION","RECALL"]});

Errors

Code	HTTP	Retryable	Description
VALIDATION_ERROR	400	No	The request payload failed schema validation.
UNAUTHORIZED	401	No	Missing or invalid API key.
SKILL_NOT_ACCESSIBLE	403	No	The API key is not granted access to this skill.
SKILL_NOT_AVAILABLE_FOR_PLAN	403	No	This skill is not included in your current plan.
PLAN_ELIGIBILITY_UNAVAILABLE	503	Yes	Plan eligibility could not be decided. Retry after backoff.
SKILL_NOT_FOUND	404	No	The requested skill or version does not exist.
TENANT_RATE_LIMITED	429	Yes	Request rate limit exceeded. Retry after backoff.
QUOTA_EXCEEDED	429	Yes	Plan quota exhausted. Retryable once the quota period resets.
EXECUTION_TIMEOUT	504	Yes	Execution exceeded the allowed time budget.
RETRIEVAL_METRICS_INVALID	400	No	The request payload failed this skill's input validation.
RETRIEVAL_METRICS_DUPLICATE_ID	400	No	The request payload failed this skill's input validation.
RETRIEVAL_METRICS_LIMIT_EXCEEDED	429	No	A size, quota, rate, or time limit was exceeded.

Other limits

Limit	Value
Recommended inline bytes	32 KiB

Limitations

Does not retrieve documents
Does not run a significance test
Does not pool judgments
Running the retriever itself
Claiming statistical significance

How to call

POST /v1/execute (requires authentication)

Request bodyjson

{
  "skillKey": "knowledge.retrieval_metrics",
  "input": {
    "queries": [
      {
        "judgments": [
          {
            "id": "d1",
            "grade": 1
          },
          {
            "id": "d3",
            "grade": 1
          }
        ],
        "id": "q1",
        "ranked": [
          "d1",
          "d2",
          "d3"
        ]
      }
    ],
    "cutoffs": [
      1,
      3
    ],
    "metrics": [
      "PRECISION",
      "RECALL"
    ]
  }
}

Examples

Binary labels yield precision 1 at cutoff 1 and recall 1 at cutoff 3

Inputjson

{
  "queries": [
    {
      "judgments": [
        {
          "id": "d1",
          "grade": 1
        },
        {
          "id": "d3",
          "grade": 1
        }
      ],
      "id": "q1",
      "ranked": [
        "d1",
        "d2",
        "d3"
      ]
    }
  ],
  "cutoffs": [
    1,
    3
  ],
  "metrics": [
    "PRECISION",
    "RECALL"
  ]
}

Expected outputjson

{
  "macro": [
    {
      "cutoff": 1,
      "precision": 1,
      "recall": 0.5
    },
    {
      "cutoff": 3,
      "precision": 0.666666667,
      "recall": 1
    }
  ],
  "queries": [
    {
      "id": "q1",
      "cutoffs": [
        {
          "cutoff": 1,
          "precision": 1,
          "recall": 0.5,
          "retrieved": 1,
          "judgedRetrieved": 1,
          "relevantRetrieved": 1
        },
        {
          "cutoff": 3,
          "precision": 0.666666667,
          "recall": 1,
          "retrieved": 3,
          "judgedRetrieved": 2,
          "relevantRetrieved": 2
        }
      ],
      "judgedCount": 2,
      "retrievedCount": 3,
      "relevantCount": 2
    }
  ],
  "unjudgedPolicy": "NON_RELEVANT",
  "formulaVersion": "knowledge.retrieval_metrics.v1"
}

Tip

d1 is relevant at rank 1; d3 is relevant at rank 3.

No relevant judgments yield recall MAP and nDCG of 0