Robots.txt Parser

Fetch and read a site's robots.txt: rules for each crawler, sitemaps, crawl delays, and whether all crawlers are blocked.

Pay by wallet (x402) robots-txt-parse

Example

Input

{
  "url": "https://strale.dev/robots.txt"
}

Answer

{
  "url": "https://strale.dev/robots.txt",
  "rules": [
    {
      "allow": [
        "/"
      ],
      "disallow": [],
      "user_agent": "Googlebot"
    },
    {
      "allow": [
        "/"
      ],
      "disallow": [],
      "user_agent": "Bingbot"
    },
    {
      "allow": [
        "/"
      ],
      "disallow": [],
      "user_agent": "Twitterbot"
    },
    {
      "allow": [
        "/"
      ],
      "disallow": [],
      "user_agent": "facebookexternalhit"
    },
    {
      "allow": [
        "/"
      ],
      "disallow": [],
      "user_agent": "*"
    }
  ],
  "exists": true,
  "sitemaps": [
    "https://strale.dev/sitemap.xml"
  ],
  "raw_length": 201,
  "total_rules": 5,
  "user_agents": [
    "Googlebot",
    "Bingbot",
    "Twitterbot",
    "facebookexternalhit",
    "*"
  ],
  "total_allow_rules": 5,
  "blocks_all_crawlers": false,
  "total_disallow_rules": 0
}

Inputs

Field	Type	Required	What it is
`url`	string	Yes	—

Output fields

How reliably each field appears in a successful answer.

Field	Type	Appears
`rules`	array	Always (present in every successful answer)
`exists`	boolean	Always (present in every successful answer)
`sitemaps`	array	Always (present in every successful answer)
`user_agents`	array	Always (present in every successful answer)
`blocks_all_crawlers`	boolean	Always (present in every successful answer)

How to call it

The same tool through each route. MCP clients reach it through Strale’s server at https://api.strale.io/mcp; the call below is what the agent sends.

# Needs an API key. max_price_cents caps what this call may cost.
curl -X POST https://api.strale.io/v1/do \
  -H "Authorization: Bearer $STRALE_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "capability_slug": "robots-txt-parse",
    "inputs": {
      "url": "https://strale.dev/robots.txt"
    },
    "max_price_cents": 2
  }'

{
  "tool": "strale_execute",
  "arguments": {
    "slug": "robots-txt-parse",
    "inputs": {
      "url": "https://strale.dev/robots.txt"
    },
    "max_price_cents": 2
  }
}

import { Strale } from "straleio";

const strale = new Strale({ apiKey: process.env.STRALE_API_KEY! });
const result = await strale.do({
  capability_slug: "robots-txt-parse",
  inputs: {
    "url": "https://strale.dev/robots.txt"
  },
  max_price_cents: 2,
});

import os
from straleio import Strale, DoRequest

strale = Strale(api_key=os.environ["STRALE_API_KEY"])
result = strale.do(DoRequest(
    capability_slug="robots-txt-parse",
    inputs={
        "url": "https://strale.dev/robots.txt",
    },
    max_price_cents=2,
))

# No account. The first request answers 402 with the price in USDC;
# an x402 client pays it from your wallet and repeats the request.
curl -i 'https://api.strale.io/x402/v2/robots-txt-parse?url=https%3A%2F%2Fstrale.dev%2Frobots.txt'

Limitations

Non-standard directives may not be interpreted
Parses standard robots.txt directives — non-standard extensions (e.g., Crawl-delay, custom directives) may be reported but not interpreted
What to do instead: Check the raw text in the response for non-standard directives that may affect your specific crawler
Only the first 500 KiB of robots.txt is parsed
Only the first 500 KiB of robots.txt is parsed. Directives beyond that limit do not appear in the result.