Test Stability Statistics
quality.flaky_test_detect
Overview
When you need to gate CI or triage unstable tests, we compute deterministic stability statistics — flip rate, pass rate, longest pass/fail runs, and transition counts — over the ordered run histories you supply. You set the thresholds; we report which tests exceed them via exceedsThresholds and publish the formulas in every response. We never label a test as flaky and we do not run tests for you — do not use this skill when you need execution, log parsing, or a definitive flaky verdict.
Stability gating needs transparent, repeatable statistics — not opaque labels that pretend to detect flakiness with certainty.
Reliability
The same history and the same thresholds always give the same numbers, and nothing leaves the platform. Every answer carries statisticsDefinition so the arithmetic is auditable, and exceedsThresholds only ever means a check you set was not met — it is never a claim that a test is flaky. Two details matter when reading the numbers: ERROR counts as a failure throughout, and a skipped run is left out entirely unless skippedPolicy is treat_as_failure, so runCount can be lower than the runs you sent while skipCount still reports them. Four failures stop the call, each needing a fix from you rather than a retry. FLAKY_TEST_LIMIT_EXCEEDED means more than 5000 tests, more than 1000 runs on one test, or more than 200000 runs altogether, so send the history in batches or shorten the window. FLAKY_RUN_SEQUENCE_INVALID means a test's sequence numbers are not 1, 2, 3 and so on with no gaps or repeats, so renumber them; order in the array does not matter, only the numbers. FLAKY_THRESHOLD_INVALID means minRuns is below 1 or a rate falls outside 0 to 1. FLAKY_OUTCOME_UNSUPPORTED means an outcome is missing or is not exactly PASS, FAIL, ERROR, or SKIP, in capitals.
Context footprint
Returns per-test statistics with thresholdOutcomes, a summary block, and the published formula definitions. A successful call costs 1 execution credit.
When to use it
When you already have ordered PASS/FAIL/ERROR/SKIP run histories and want flip rate, pass rate, and threshold comparisons you can audit.
When you need test execution, log parsing, or a definitive flaky verdict.
Scope
Non-goals
- Does not assert flakiness
- Does not execute tests
- Does not parse logs
- Does not predict future failures
Properties
Capabilities
Cost & billing
Flat rate per invocation
File handling
How we handle your data
What this means
This skill may receive sensitive content, but it does not target sensitive data and returns only a structured non-sensitive result.
Squillify guarantees
- Runs inside Squillify
- This skill runs inside Squillify.
- Payloads are not sent to an external service by this skill.
- Squillify does not permanently store submitted payloads.
- Telemetry records metadata only, not submitted content.
- Submitted payloads are not used for training.
- Submitted payloads are not reviewed by humans.
- This skill does not target sensitive data.
- The response does not contain submitted content.
Your responsibility
Squillify does not determine whether submitted content is sensitive. Your application is responsible for deciding what data it is allowed to submit. Your application is responsible for securing the response.
Input schema
| Field | Type | Required | Description |
|---|---|---|---|
| tests | array | Yes | Test histories to score, 1 through 5000 closed {id, runs} objects and at most 200000 runs in total. |
| thresholds | object | No | Closed optional bars compared against computed statistics; omit a field to skip that comparison, or omit the object to skip all. |
| options | object | No | Closed analysis options for skipped runs and sort order; omit the object to ignore SKIP runs and sort by flip rate descending. |
Output schema
| Field | Type | Required | Description |
|---|---|---|---|
| tests | array | Yes | Statistics rows, 1 through 5000 closed objects ordered by flipRate descending then id. |
| summary | object | Yes | Closed request totals with testCount, exceedingCount, insufficientDataCount and totalObservations. |
| statisticsDefinition | string | Yes | Published one-line formulas behind every reported rate and streak. |
| findings | array | Yes | Request-wide notices, 0 through 100 strings; always empty today. |
| _meta | object | Yes | Closed bookkeeping of the skipped-run policy and sort order that produced these numbers. |
API contract
POST /v1/executecurl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"quality.flaky_test_detect","input":{"tests":[{"id":"com.example.SampleTest#runs","runs":[{"sequence":1,"outcome":"PASS"}]}]}}'import { Squillify } from "@squillify/sdk";
const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });
const { output } = await sq.execute("quality.flaky_test_detect", {"tests":[{"id":"com.example.OrderTest#createsOrder","runs":[{"outcome":"PASS","sequence":1},{"outcome":"FAIL","sequence":2},{"outcome":"PASS","sequence":3}]}],"options":{"skippedPolicy":"ignore","sort":"flip_rate_desc"},"thresholds":{"minRuns":5,"maxFlipRate":0.1}});Errors
| Code | HTTP | Retryable | Description |
|---|---|---|---|
| VALIDATION_ERROR | 400 | No | The request payload failed schema validation. |
| UNAUTHORIZED | 401 | No | Missing or invalid API key. |
| SKILL_NOT_ACCESSIBLE | 403 | No | The API key is not granted access to this skill. |
| SKILL_NOT_AVAILABLE_FOR_PLAN | 403 | No | This skill is not included in your current plan. |
| PLAN_ELIGIBILITY_UNAVAILABLE | 503 | Yes | Plan eligibility could not be decided. Retry after backoff. |
| SKILL_NOT_FOUND | 404 | No | The requested skill or version does not exist. |
| TENANT_RATE_LIMITED | 429 | Yes | Request rate limit exceeded. Retry after backoff. |
| QUOTA_EXCEEDED | 429 | Yes | Plan quota exhausted. Retryable once the quota period resets. |
| EXECUTION_TIMEOUT | 504 | Yes | Execution exceeded the allowed time budget. |
| FLAKY_TEST_LIMIT_EXCEEDED | 429 | No | A size, quota, rate, or time limit was exceeded. |
| FLAKY_RUN_SEQUENCE_INVALID | 400 | No | The request payload failed this skill's input validation. |
| FLAKY_THRESHOLD_INVALID | 400 | No | The request payload failed this skill's input validation. |
| FLAKY_OUTCOME_UNSUPPORTED | 400 | No | The skill does not support the requested variant. |
Other limits
| Limit | Value |
|---|---|
| Recommended inline bytes | 32 KiB |
Limitations
- Does not assert flakiness
- Does not execute tests
- Does not parse logs
- Does not predict future failures
- Proving a test is flaky
- Root-cause analysis of failures
- Live test execution or log ingestion
How to call
POST /v1/execute (requires authentication)
{
"skillKey": "quality.flaky_test_detect",
"input": {
"tests": [
{
"id": "com.example.OrderTest#createsOrder",
"runs": [
{
"outcome": "PASS",
"sequence": 1
},
{
"outcome": "FAIL",
"sequence": 2
},
{
"outcome": "PASS",
"sequence": 3
}
]
}
],
"options": {
"skippedPolicy": "ignore",
"sort": "flip_rate_desc"
},
"thresholds": {
"minRuns": 5,
"maxFlipRate": 0.1
}
}
}Examples
Gate a test that exceeds your flip-rate threshold
{
"tests": [
{
"id": "com.example.OrderTest#createsOrder",
"runs": [
{
"outcome": "PASS",
"sequence": 1
},
{
"outcome": "FAIL",
"sequence": 2
},
{
"outcome": "PASS",
"sequence": 3
}
]
}
],
"options": {
"skippedPolicy": "ignore",
"sort": "flip_rate_desc"
},
"thresholds": {
"minRuns": 5,
"maxFlipRate": 0.1
}
}{
"_meta": {
"sort": "flip_rate_desc",
"skippedPolicy": "ignore"
},
"statisticsDefinition": "flipRate = transitions / (runCount - 1); passRate = passCount / evaluatedRunCount",
"summary": {
"testCount": 1,
"exceedingCount": 1,
"insufficientDataCount": 1,
"totalObservations": 3
},
"tests": [
{
"runCount": 3,
"lastOutcome": "PASS",
"flipRate": 1,
"insufficientData": true,
"id": "com.example.OrderTest#createsOrder",
"longestFailureRun": 1,
"flipCount": 2,
"exceedsThresholds": true,
"thresholdOutcomes": [
{
"met": false,
"limit": 5,
"value": 3,
"threshold": "minRuns"
},
{
"met": false,
"limit": 0.1,
"value": 1,
"threshold": "maxFlipRate"
}
],
"failCount": 1,
"passRate": 0.6667,
"skipCount": 0,
"passCount": 2,
"longestPassRun": 1
}
],
"findings": []
}Confirm a stable suite meets your thresholds
{
"thresholds": {
"maxFlipRate": 0.1,
"minPassRate": 0.95,
"minRuns": 5
},
"tests": [
{
"id": "com.example.CheckoutTest#pays",
"runs": [
{
"outcome": "PASS",
"sequence": 1
},
{
"outcome": "PASS",
"sequence": 2
},
{
"outcome": "PASS",
"sequence": 3
},
{
"outcome": "PASS",
"sequence": 4
},
{
"outcome": "PASS",
"sequence": 5
}
]
}
]
}{
"_meta": {
"sort": "flip_rate_desc",
"skippedPolicy": "ignore"
},
"statisticsDefinition": "flipRate = transitions / (runCount - 1); passRate = passCount / evaluatedRunCount",
"summary": {
"testCount": 1,
"exceedingCount": 0,
"insufficientDataCount": 0,
"totalObservations": 5
},
"tests": [
{
"runCount": 5,
"lastOutcome": "PASS",
"flipRate": 0,
"insufficientData": false,
"id": "com.example.CheckoutTest#pays",
"longestFailureRun": 0,
"flipCount": 0,
"exceedsThresholds": false,
"thresholdOutcomes": [
{
"met": true,
"limit": 5,
"value": 5,
"threshold": "minRuns"
},
{
"met": true,
"limit": 0.1,
"value": 0,
"threshold": "maxFlipRate"
},
{
"met": true,
"limit": 0.95,
"value": 1,
"threshold": "minPassRate"
}
],
"failCount": 0,
"passRate": 1,
"skipCount": 0,
"passCount": 5,
"longestPassRun": 5
}
],
"findings": []
}Rank two tests by flip rate for triage
{
"tests": [
{
"id": "b.payment",
"runs": [
{
"outcome": "PASS",
"sequence": 1
},
{
"outcome": "FAIL",
"sequence": 2
}
]
},
{
"id": "a.checkout",
"runs": [
{
"outcome": "PASS",
"sequence": 1
},
{
"outcome": "FAIL",
"sequence": 2
},
{
"outcome": "PASS",
"sequence": 3
}
]
}
],
"options": {
"sort": "flip_rate_desc"
},
"thresholds": {
"minRuns": 1
}
}{
"_meta": {
"sort": "flip_rate_desc",
"skippedPolicy": "ignore"
},
"statisticsDefinition": "flipRate = transitions / (runCount - 1); passRate = passCount / evaluatedRunCount",
"summary": {
"testCount": 2,
"exceedingCount": 0,
"insufficientDataCount": 0,
"totalObservations": 5
},
"tests": [
{
"runCount": 3,
"lastOutcome": "PASS",
"flipRate": 1,
"insufficientData": false,
"id": "a.checkout",
"longestFailureRun": 1,
"flipCount": 2,
"exceedsThresholds": false,
"thresholdOutcomes": [
{
"met": true,
"limit": 1,
"value": 3,
"threshold": "minRuns"
}
],
"failCount": 1,
"passRate": 0.6667,
"skipCount": 0,
"passCount": 2,
"longestPassRun": 1
},
{
"runCount": 2,
"lastOutcome": "FAIL",
"flipRate": 1,
"insufficientData": false,
"id": "b.payment",
"longestFailureRun": 1,
"flipCount": 1,
"exceedsThresholds": false,
"thresholdOutcomes": [
{
"met": true,
"limit": 1,
"value": 2,
"threshold": "minRuns"
}
],
"failCount": 1,
"passRate": 0.5,
"skipCount": 0,
"passCount": 1,
"longestPassRun": 1
}
],
"findings": []
}Recommended use cases
- Rank tests by flip rate before triage
- Gate CI merges on stability thresholds you define
- Summarize historical pass/fail patterns from exported run data
Avoid for
- Proving a test is flaky
- Root-cause analysis of failures
- Live test execution or log ingestion
Related skills
Composable with
Safety & privacy
- Runs without calling any external service.
- Your payloads are never stored permanently.
- Telemetry captures metadata only — never your payload contents.
- We never train models on your payloads.
- No human reviews your payloads.