How do I connect an AI agent to Robots.txt & AI Crawler Checker?

Use the MCP streamable HTTP endpoint https://robots-check.quietworklabs.workers.dev/mcp.

Catalog › dev.workers.quietworklabs.robots-check/robots-check

Is Robots.txt & AI Crawler Checker reliable?

Reliable (measured) Yes. Faceabot measured Robots.txt & AI Crawler Checker 4 times: 100% of checks succeeded, including the last 4 in a row (last check 2026-10-19 20:00 UTC). It also does the job: its real read-only task worked 4 of 4 times.

Last check: 2026-10-19 20:00 UTC · answered in 5701 ms · remote (no install)

En français : Fiable (mesuré). Faceabot teste cet outil lui-même, en continu, sans dépendre de ce que son auteur affirme.

What it does

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Source: official MCP registry (registry.modelcontextprotocol.io) · version 1.0.0 · publisher website

Connect your agent

{
  "mcpServers": {
    "robots-check": {
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

Or ask Faceabot first: MCP tool check_reliability {"id":"dev.workers.quietworklabs.robots-check/robots-check"} on https://faceabot.com/api/mcp.

Does it actually do the job?

Does the job (tested) — Faceabot asked ai_crawler_access to do a small read-only task 4 time(s): 4 returned a real result, the last 4 in a row. Last result: ok in 3748 ms.

Sample input: {"url":"https://www.theguardian.com/"}. A success means a non-empty result without error; it does not certify the accuracy or safety of the content.

Faceabot measurements

When	Answers	Real task	Latency
2026-10-19 20:00 UTC	✅ ok	✅ done	5701 ms
2026-10-19 06:45 UTC	✅ ok	✅ done	7804 ms
2026-10-18 14:20 UTC	✅ ok	✅ done	6824 ms
2026-10-18 04:05 UTC	✅ ok	✅ done	5313 ms

Method: MCP initialize then tools/list from Faceabot (FaceabotProbe/1.0), plus one read-only tool call with a neutral input when the server declares one, repeated over time. This checks that the server is up and speaks MCP correctly; it does not certify the quality or safety of each tool. How measurement works.

Publisher of Robots.txt & AI Crawler Checker?

Show your measured reliability on your README or website:

[![Measured by Faceabot](https://faceabot.com/badge/dev.workers.quietworklabs.robots-check/robots-check.svg)](https://faceabot.com/tools/dev.workers.quietworklabs.robots-check/robots-check)

Data in JSON: /api/catalog?id=dev.workers.quietworklabs.robots-check/robots-check · Stop probes: POST /api/reliability/opt-out

AI agents: all of this is available without keys over MCP https://faceabot.com/api/mcp · HTTP https://faceabot.com/api/catalog?q=… · Guide : llms.txt

Best tool by topic · Pulse: what is changing in agent commerce · Is your store ready for AI shopping agents? · State of MCP (weekly) · MCP server doctor · Reliability data · Faceabot app · Faceabot → AI assistants · What AI agents cannot find yet · Add Faceabot to your AI · Badge for tool publishers · Help · Privacy · Terms