Google tells you what exists. Faceabot tells you what holds up.
Checks if a URL path is allowed by robots.txt. x402 payment required (testnet USDC).
MCP endpoint :https://faceted-napped-debtless.ngrok-free.dev/robots-txt-checker/mcp · measurements and detailsChecks llms.txt, AI crawler access in robots.txt, and sitemap - with a 0-100 AI readiness score.
MCP endpoint :https://mcp.apify.com?tools=apifmcpfactory/llms-txt-checker · website · measurements and detailsAI visibility checker: GPTBot vs OAI-SearchBot, robots.txt audit, raw-HTML schema. 165 crawlers.
Package mcpb :https://github.com/krisdiallo/ecom-agent/releases/download/v1.5.3/aivis.mcpb · website · measurements and detailsrobots.txt group count
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-count/mcp · website · measurements and detailsrobots.txt Allow count, body discarded
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp · website · measurements and details{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
MCP endpoint :https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots · website · measurements and detailsFetch a public robots.txt and return group counts only.
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-inspect/mcp · website · measurements and detailsChecks robots.txt line by line and says what each AI crawler token actually controls - training, AI
Package npm :@readystack/robots-txt-ai-crawler-audit · website · measurements and detailsCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
MCP endpoint :https://actablesite.com/mcp/ · website · measurements and detailsURL to clean markdown for LLMs: a polite, robots.txt-respecting web reader. Free, no API key
MCP endpoint :https://agent-web.foomworks.workers.dev/mcp · website · measurements and detailsllms.txt 檢查工具：檢查網站的 llms.txt、robots 與 sitemap。台灣繁體中文 MCP 工具。
MCP endpoint :https://llms.penguindriver.com/mcp · website · measurements and detailsDelegate 20 typed RQM Robotics jobs for validation, diagnosis, comparison, and optimization.
MCP endpoint :https://jobs.rqmtechnologies.com/mcp/robotics · website · measurements and details檢查網站 llms.txt 與 AI 爬蟲 robots.txt 設定，並自動產生 llms.txt。
MCP endpoint :https://llms-check.panda198271.workers.dev/mcp · website · measurements and detailsMarket & supply-chain intelligence on 34 humanoid robots: scores, specs, pricing. No key.
MCP endpoint :https://mcp-free.roboselect360.com/mcp · measurements and detailsSEO tools for AI agents: audits, Core Web Vitals, keywords, AI visibility, traffic, backlinks.
MCP endpoint :https://www.robot-speed.com/api/mcp · website · measurements and detailsCheckout-verified prices, stock, US FCC status and buy routes for humanoid and quadruped robots
MCP endpoint :https://mcp.botinfo.ai/mcp · website · measurements and detailsVendor-neutral catalog of 300+ commercial robots: search, compare specs, request vendor intros.
MCP endpoint :https://robohub.app/api/mcp · website · measurements and detailsFX and metals market data: plans, 51 symbols, price quotes, and candles with an API key.
MCP endpoint :https://api.adona-robot.com/mcp · website · measurements and detailsDoes a site publish llms.txt, robots permissions and a sitemap agents can use? Bounded check.
MCP endpoint :https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/llms-txt-readiness/mcp · website · measurements and detailsToken compression for web content — 70-80% reduction from HTML to clean markdown
Package npm :@robot-resources/scraper-mcp · website · measurements and detailsAI agents: all of this is available without keys over MCP https://faceabot.com/api/mcp · HTTP https://faceabot.com/api/catalog?q=… · Guide : llms.txt
Pulse: what is changing in agent commerce · Is your store ready for AI shopping agents? · State of MCP (weekly) · MCP server doctor · Open reliability data · Add Faceabot to your AI · Help · Privacy · Terms