AI Crawler Access Checker
Before ChatGPT, Claude or Perplexity can mention your brand, their crawlers have to reach your site. This tool sends 20+ real AI & search crawler user-agents at your URL and checks robots.txt rules plus live HTTP access — so you know exactly who can (and can't) see you.
We fetch your robots.txt and probe each crawler's user-agent against your live URL. No data is stored.
Crawler access is one of four signals we score. The full audit also checks AI model recall, structured data and crawl budget, then returns a single graded report.
Run the full auditCrawler access reference
Blocking an AI crawler is the one visibility mistake that is both invisible in analytics and total in effect. If GPTBot cannot fetch your pages, no amount of content or markup will get you cited in ChatGPT. This tool resolves robots.txt rules per user-agent and probes your live URL with each crawler's real UA string.
What each crawler is for
| User-agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training corpus collection |
| OAI-SearchBot | OpenAI | ChatGPT search index |
| ChatGPT-User | OpenAI | Live fetch when a user shares a link |
| ClaudeBot | Anthropic | Training corpus collection |
| Claude-Web | Anthropic | Live fetch during a conversation |
| PerplexityBot | Perplexity | Answer engine index |
| Google-Extended | Gemini training opt-out token | |
| CCBot | Common Crawl | Open corpus most models train on |
The distinction that costs people visibility
Training crawlers and answer-engine crawlers are separate user-agents. Blocking GPTBot keeps your content out of future training runs but leaves OAI-SearchBot free to index you for ChatGPT search — that is usually the combination you want. Blocking both removes you from ChatGPT entirely. Google-Extended is not a crawler at all: it is a policy token that controls Gemini training use without affecting Google Search ranking.
How the score is calculated
The score is the share of AI crawlers allowed by robots.txt for the specific path you tested. robots.txt is evaluated with real matching semantics: the most specific User-agent group wins, the longest matching Disallow or Allow path takes precedence, and a missing robots.txt means everything is permitted.
Reading the probe result honestly
We report robots.txt permission and a live fetch separately. A 403 or 429 on our probe usually means your WAF or rate limiter rejected our datacenter IP, not that the crawler is blocked — real crawler IPs are often allowlisted. Cloudflare's AI bot management, for instance, blocks by IP reputation and behaviour, which no external tool can fully replicate. When robots.txt says allowed but our probe fails, the note explains the difference rather than reporting a false failure.
Fixing a block
- Check for a broad
User-agent: *withDisallow: /left over from a staging config. - Look for Cloudflare's "Block AI Scrapers and Crawlers" toggle, which injects blocks independent of your robots.txt file.
- Verify your CDN or WAF is not challenging non-browser user-agents by default.
- Re-test the specific path, not just the homepage. Rules are path-scoped.
Access is only the entry ticket. Recall, markup and crawl budget decide whether you actually get cited.
Check all four signals at once