FaceabotFREN
‹ Back

The real catalog of tools for AI agents

Google tells you what exists. Faceabot tells you what holds up.

44,737real tools
29,284callable over HTTPS
19,887measured
6,168proven reliable
Robots.txt & AI Crawler Checker Reliable (measured) · 100% over 4 checks Does the job (tested) · 4/4

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

MCP endpoint : https://robots-check.quietworklabs.workers.dev/mcp · website · measurements and details
polite-crawler Not measured yet

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

Package npm : polite-crawler-mcp · website · measurements and details
agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free) Reliable (measured) · 100% over 42 checks

URL to clean markdown for LLMs: a polite, robots.txt-respecting web reader. Free, no API key

MCP endpoint : https://agent-web.foomworks.workers.dev/mcp · website · measurements and details
fastCRW Private (key required)

Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.

MCP endpoint : https://fastcrw.com/mcp · website · measurements and details
AI Crawler Access Check Flaky · 94% over 18 checks Partly does the job · 13/17

Can GPTBot, ClaudeBot and PerplexityBot reach a site? Real fetch per crawler, firewall/CDN blocks.

MCP endpoint : https://check.citant.ai/mcp?src=registry · website · measurements and details
UAParse Reliable (measured) · 100% over 38 checks

Parse a User-Agent: browser, OS, device, plus bot, search crawler, AI crawler and AI agent flags.

MCP endpoint : https://uaparse.mike-tusa.workers.dev/mcp · website · measurements and details
seo-crawler-mcp Not measured yet

Crawl and analyse websites for SEO errors using Crawlee with SQLite storage

Package npm : @houtini/seo-crawler-mcp · website · measurements and details
AI Crawler Index Reliable (measured) · 100% over 42 checks Does the job (tested) · 42/42

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

MCP endpoint : https://www.pathwren.workers.dev/c/mcp-registry-official/mcp · website · measurements and details
Crawler IP Verifier - real Googlebot or fake Flaky · 75% over 4 checks Does the job (tested) · 3/3

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

MCP endpoint : https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/netcheck · website · measurements and details
ai-visibility Reliable (measured) · 100% over 43 checks

AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.

MCP endpoint : https://ai-visibility.lastminutedealshq.com/mcp · website · measurements and details
oassis — web scraping, crawling and browser control for AI agents Flaky · 88% over 8 checks Fails real tasks · 0/7

Scrape to markdown, map and crawl sites, drive a real browser. Pay per call, no API key, no signup.

MCP endpoint : https://api.oassis.dev/mcp · website · measurements and details
IndexPulse Private (key required)

Bing Webmaster, IndexNow and Google Search Console for AI: rankings, indexing gaps, AI crawlers.

MCP endpoint : https://indexpulse.puzaricloud.in/mcp · website · measurements and details
AI Visibility Audit — Can AI crawlers read and cite your site? Reliable (measured) · 100% over 43 checks

One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.

MCP endpoint : https://ai-visibility.rowb.app/mcp · website · measurements and details
smart-crawler Private (key required)

E-commerce competitor intelligence, VOC sentiment, Amazon ASIN analysis, Reddit growth playbooks.

MCP endpoint : https://smartcrawler.io/mcp/ · website · measurements and details
ai-crawler-readiness Private (key required)

Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index

MCP endpoint : https://shop.lumnika.com/ai-readiness/mcp · website · measurements and details
TidyTools: leads, fetch to Markdown, ads, domain authority, jobs, prices, AI crawlers Private (key required)

Pay-per-use web data tools: leads, fetch to Markdown, ads, domain authority, jobs, prices.

MCP endpoint : https://mcp.apify.com/?tools=tidytools/company-website-enrichment,tidytools/website-contact-extractor,tidytools/website-markdown-crawler,tidytools/document-to-markdown,tidytools/audio-transcriber,tidytools/ai-crawler-access-checker,tidytools/smart-fetch,tidytools/regional-price-check,tidytools/places-database,tidytools/ats-job-search,tidytools/ads-transparency,tidytools/domain-authority · website · measurements and details
ActableSite AI Crawler Monitor Flaky · 86% over 7 checks Does the job (tested) · 6/6

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

MCP endpoint : https://actablesite.com/mcp/ · website · measurements and details
Unrendered: AI Crawler Checker Not measured yet

AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.

Package npm : unrendered-mcp · website · measurements and details
ai-crawler-policy Reliable (measured) · 100% over 41 checks

Cloudflare Workers MCP server: ai-crawler-policy

MCP endpoint : https://api.lazy-mac.com/ai-crawler-policy/mcp · website · measurements and details
Atlas Web Compiler & Scraper Private (key required)

Deterministic Zero-LLM Web Compiler, Scraper, Crawler, and Site Mapper for AI Agents.

MCP endpoint : https://api.atlas-compiler.com/mcp · website · measurements and details

AI agents: all of this is available without keys over MCP https://faceabot.com/api/mcp · HTTP https://faceabot.com/api/catalog?q=… · Guide : llms.txt

Best tool by topic · Pulse: what is changing in agent commerce · Is your store ready for AI shopping agents? · State of MCP (weekly) · MCP server doctor · Reliability data · Faceabot app · Faceabot → AI assistants · What AI agents cannot find yet · Add Faceabot to your AI · Badge for tool publishers · Help · Privacy · Terms