Parse robots.txt

Overview

You hand us the text of a robots.txt file and we split it into user-agent groups, work out whether one path is crawlable by one agent, and collect the Sitemap lines. Matching follows RFC 9309: the longest matching rule wins, and Allow beats Disallow when two rules tie on length. We only read the content you send, never the network, and the answer is crawl etiquette rather than an access control you can rely on.

Robots rules look simple and then bite: longest-match and tie-breaking are easy to get wrong by hand, and you want the same answer every time you ask.

When to use it

Scope

Non-goals

• Does not fetch robots.txt, or anything else, over the network
• Does not crawl, queue, or schedule crawls
• Does not percent-decode paths before matching them
• Does not honour Crawl-delay, Host, or other non-standard lines
• Is crawl etiquette, never an authorization or security control

Properties

Capabilities

Cost & billing

Flat rate per invocation

File handling

How we handle your data

Input schema

Output schema

API contract

curl -sS -X POST "$BASE/v1/execute" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"skillKey":"web.parse_robots_txt","input":{"content":"User-agent: *\nDisallow: /private","path":"/private"}}'

import { Squillify } from "@squillify/sdk";

const sq = new Squillify({ apiKey: process.env.SQUILLIFY_API_KEY });

const { output } = await sq.execute("web.parse_robots_txt", {"path":"/private/page.html","content":"User-agent: *\nDisallow: /private\n","userAgent":"SquillifyBot"});

Errors

Other limits

Limitations

• Does not fetch robots.txt, or anything else, over the network
• Does not crawl, queue, or schedule crawls
• Does not percent-decode paths before matching them
• Does not honour Crawl-delay, Host, or other non-standard lines
• Is crawl etiquette, never an authorization or security control
• Fetching robots.txt for a domain — bring the text with you instead
• Deciding who may reach a resource, which needs a real access control

How to call

POST /v1/execute (requires authentication)

{
  "skillKey": "web.parse_robots_txt",
  "input": {
    "path": "/private/page.html",
    "content": "User-agent: *\nDisallow: /private\n",
    "userAgent": "SquillifyBot"
  }
}

Examples

{
  "path": "/private/page.html",
  "content": "User-agent: *\nDisallow: /private\n",
  "userAgent": "SquillifyBot"
}

{
  "warnings": [],
  "matchedRule": {
    "directive": "Disallow",
    "pattern": "/private",
    "line": 2
  },
  "sitemaps": [],
  "allowed": false,
  "groups": [
    {
      "rules": [
        {
          "directive": "Disallow",
          "line": 2,
          "value": "/private"
        }
      ],
      "userAgents": [
        "*"
      ]
    }
  ]
}

{
  "path": "/private/public/page",
  "content": "User-agent: *\nDisallow: /private\nAllow: /private/public\n"
}

{
  "warnings": [],
  "matchedRule": {
    "directive": "Allow",
    "pattern": "/private/public",
    "line": 3
  },
  "sitemaps": [],
  "allowed": true,
  "groups": [
    {
      "rules": [
        {
          "directive": "Disallow",
          "line": 2,
          "value": "/private"
        },
        {
          "directive": "Allow",
          "line": 3,
          "value": "/private/public"
        }
      ],
      "userAgents": [
        "*"
      ]
    }
  ]
}

Recommended use cases

• Check a path is crawlable before you queue it
• Pull the sitemap URLs a site declares in its robots.txt
• Audit the crawl rules in a saved robots.txt snapshot

Avoid for

• Fetching robots.txt for a domain — bring the text with you instead
• Deciding who may reach a resource, which needs a real access control

Related skills

Composable with

Safety & privacy

• Runs without calling any external service.
• Your payloads are never stored permanently.
• Telemetry captures metadata only — never your payload contents.
• We never train models on your payloads.
• No human reviews your payloads.

Tags