Free Robots.txt AI Bot Checker: see which AI crawlers reach your site
Paste your domain or robots.txt URL. We read the file, match every major AI crawler against your Allow and Disallow rules, and tell you bot by bot which ones reach your site. No sign-up needed.
Free, no sign-up. We read your robots.txt and show which AI crawlers — GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more — you currently allow or block.
What the AI bot checker validates
- Per-bot verdict (Allowed, Partial, Blocked) for every major AI user-agent
- Training crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Meta-ExternalAgent
- Search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot-Extended
- On-demand fetch bots: ChatGPT-User, Claude-User, Perplexity-User
- Robots.txt URL location and HTTP status code
- Conflicts between User-agent: * and bot-specific rules
- Disallowed paths per bot — so you can see what's hidden, not just what's blocked
Why your robots.txt matters more than it used to
Your robots.txt is the first thing a crawler reads, and AI companies now run multiple crawlers with separate user-agents. OpenAI alone has three: GPTBot for training, OAI-SearchBot for ChatGPT Search, and ChatGPT-User for on-demand fetches. Anthropic, Google, Perplexity, Common Crawl and Meta each have their own.
The decisions you make in this file shape two different outcomes. Block the search crawlers and your brand disappears from ChatGPT, Claude and Perplexity answers. Leave the training crawlers open when you meant to opt out, and tomorrow's models learn from your content for free.
Most sites end up wanting one and not the other.
What the data says about blocking AI bots
Three numbers worth knowing before you write the rules.
How to read your result
Allowed
The crawler can reach your site root. For search bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot, this is what keeps you eligible to be cited in AI answers.
Partial
The crawler can reach your site but specific paths are disallowed. Usually fine — worth confirming you're not hiding pages you want surfaced in AI search.
Blocked
A Disallow: / rule stops the crawler entirely. Intentional for training opt-out, but a problem if it's a search crawler you wanted to stay visible to.
The most common robots.txt mistakes (and how to fix them)
Blocking AI search by accident.
A blanket Disallow that catches OAI-SearchBot, Claude-SearchBot or PerplexityBot quietly removes you from AI answers.
Fix: List the search crawlers in their own User-agent groups and allow them explicitly.
Using User-agent: * to block AI.
That group also catches Googlebot, Bingbot and every regular search crawler, so the rule blocks far more than you intended.
Fix: Name each AI user-agent in its own group and leave the wildcard alone.
Confusing Google-Extended with Googlebot.
Block the wrong one and you either leave AI training on or accidentally deindex yourself from Google Search.
Fix: Use Google-Extended to opt out of Gemini and Vertex training; leave Googlebot untouched for organic Search.
Treating robots.txt as a firewall.
It's advisory. Well-behaved crawlers honour it; bad actors and third-party scrapers ignore it.
Fix: Add server-side blocking (Cloudflare AI rules, WAF, .htaccess) for bots you must hard-stop.
Cloudflare overriding your Allow rule.
Since July 2025, Cloudflare blocks AI bots by default on new domains, regardless of what your robots.txt says.
Fix: Open the Cloudflare dashboard, go to Security → Bots → AI Scrapers and Crawlers, and switch it off if you want AI search bots to reach your site.
Want to see if AI is actually citing you?
A robots.txt audit tells you which AI crawlers can reach your site. The SEOcrawl AI Tracker tells you what they do once they're there. Learn how often ChatGPT, Claude, Gemini and Perplexity mention and cite your brand, which prompts trigger you, and how you compare against the competitors you choose to monitor — paired with your Google Search Console data on the same dashboard.
FAQs
What is an AI bot checker?
A tool that reads your robots.txt and tells you which AI crawlers you allow or block, one bot at a time. It compares your Allow and Disallow rules against the user-agent tokens of major AI companies: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Google (Google-Extended), Perplexity (PerplexityBot), Meta, Common Crawl and others.
What is GPTBot?
GPTBot is OpenAI's web crawler, launched in August 2023, used to collect publicly available content for training future GPT models. It's not the same bot as OAI-SearchBot, which powers ChatGPT Search, or ChatGPT-User, which fetches a page when someone asks an assistant to read it. All three obey robots.txt independently, so you can allow one and block another.
What's the difference between training, search and on-demand AI bots?
Training bots (GPTBot, CCBot, Google-Extended, Bytespider) scrape content to train the underlying models. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index your site so it can be cited in AI search answers. On-demand fetch bots (ChatGPT-User, Claude-User, Perplexity-User) retrieve a single page in real time when a user asks the assistant about it. Each can be controlled separately in robots.txt.
How do I block AI crawlers in robots.txt?
Add one group per crawler with a Disallow rule. For example: "User-agent: GPTBot" followed by "Disallow: /" on the next line. To block several, list each user-agent in its own group. Remember that robots.txt is advisory: well-behaved crawlers like GPTBot and ClaudeBot honour it, but it won't stop a bot that chooses to ignore it.
Should I block AI bots or allow them?
It depends on your goal. Allow the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) so your brand can be cited in AI answers. Block the training bots (GPTBot, CCBot, Google-Extended, Bytespider) only if you have a specific reason to opt out. The Rutgers/Wharton study found publishers that blocked AI crawlers wholesale lost 23% of their total traffic, so blanket bans rarely pay off.
Does ClaudeBot respect robots.txt?
Yes. Anthropic's official documentation confirms that ClaudeBot, Claude-User and Claude-SearchBot all honour standard robots.txt directives, including the non-standard Crawl-delay extension. Each user-agent can be allowed or blocked independently, so you can opt out of training while staying citable in Claude's search answers.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls whether your content is used to train and ground Gemini and Vertex AI. It's a separate token from Googlebot, so blocking Google-Extended has no effect on how you rank in Google Search. It's the clean way to opt out of AI training without touching organic search.
Will Cloudflare's default AI bot block override my robots.txt Allow rule?
Yes, network-level blocking wins. Since July 1, 2025, Cloudflare blocks AI bots by default on every new domain via its "AI Scrapers and Crawlers" managed rule, even if your robots.txt says Allow. If a bot shows up as Allowed in this checker but can't actually fetch your pages, check your Cloudflare Security → Bots settings.
