Google tells you what exists. Faceabot tells you what holds up.
robots.txt Allow count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp · website · measurements and detailsrobots.txt group count
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-count/mcp · website · measurements and detailsFetch a public robots.txt and return group counts only.
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-inspect/mcp · website · measurements and detailsChecks if a URL path is allowed by robots.txt. x402 payment required (testnet USDC).
MCP endpoint :https://faceted-napped-debtless.ngrok-free.dev/robots-txt-checker/mcp · measurements and details{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
MCP endpoint :https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots · website · measurements and detailsAllow method token count, value discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/allow-ok/mcp · website · measurements and detailsrequirements.txt name count, versions discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/requirements-ok/mcp · website · measurements and detailsCSV row count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/csv-row-count/mcp · website · measurements and detailsSitemap URL count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/sitemap-url-count/mcp · website · measurements and detailsTOML table count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/toml-table-count/mcp · website · measurements and detailsXML tag count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/xml-tag-count/mcp · website · measurements and detailsYAML key count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/yaml-key-count/mcp · website · measurements and detailsChecks robots.txt line by line and says what each AI crawler token actually controls - training, AI
Package npm :@readystack/robots-txt-ai-crawler-audit · website · measurements and detailscsproj package reference count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/csproj-count/mcp · website · measurements and detailsABNF rule count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/abnf-ok/mcp · website · measurements and detailsAsyncAPI channel count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/asyncapi-ok/mcp · website · measurements and detailsBBCode tag count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/bbcode-ok/mcp · website · measurements and detailsBNF production count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/bnf-ok/mcp · website · measurements and detailsCompose service count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/compose-ok/mcp · website · measurements and detailsCSS rule count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/css-ok/mcp · website · measurements and detailsAI agents: all of this is available without keys over MCP https://faceabot.com/api/mcp · HTTP https://faceabot.com/api/catalog?q=… · Guide : llms.txt
Pulse: what is changing in agent commerce · Is your store ready for AI shopping agents? · State of MCP (weekly) · MCP server doctor · Open reliability data · Add Faceabot to your AI · Help · Privacy · Terms