Is Your Site Blocking AI Crawlers? How to Check

To check if you're blocking AI crawlers, open yourdomain.com/robots.txt and look for Disallow: / lines under user-agents like GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot. Also check your CDN or firewall — many hosts block AI bots by default, even when robots.txt allows them.

Which AI crawlers visit my website?

A handful of named bots now crawl the web specifically for AI products. They split into two jobs: training (gathering data to build models) and search/citation (fetching live pages to answer a user's question right now). That difference matters, because the search bots are the ones that can send you traffic and get your brand named in an answer.

The main AI crawlers as of 2026:

Crawler	Who runs it	Job	Blocking it means
GPTBot	OpenAI	Trains ChatGPT models	Your content won't be used to train future models
OAI-SearchBot	OpenAI	Fetches pages for ChatGPT search citations	You can't be cited in ChatGPT's live answers
ClaudeBot	Anthropic	Trains Claude models	Your content won't be used in Claude training
Claude-SearchBot	Anthropic	Fetches pages for Claude's live answers	You can't be cited in Claude's answers
PerplexityBot	Perplexity	Indexes pages for Perplexity answers	You disappear from Perplexity citations
Google-Extended	Google	Gemini model training only	No effect on Google Search or AI Overviews
CCBot	Common Crawl	Open dataset used by many AI labs	Removes you from a widely-reused training corpus

Note the key one people get wrong: Google-Extended controls Gemini training, not Google Search or AI Overviews. Blocking Google-Extended will not remove you from AI Overviews — those are served by normal Googlebot. If you block Googlebot, that's a different (and much bigger) problem.

How do I check if I'm accidentally blocking GPTBot or ClaudeBot?

There are three places a bot can be blocked. Check all three — robots.txt is only the first.

1. Read your robots.txt. Go to https://yourdomain.com/robots.txt in a browser. Look for any block named user-agent followed by Disallow: /. A line like User-agent: GPTBot then Disallow: / blocks GPTBot from the whole site.
2. Check your CDN, WAF or host. Cloudflare, Fastly, AWS, and several managed hosts now ship "block AI bots" toggles, sometimes on by default. This is the most common silent blocker — your robots.txt says "allow" but the firewall returns a 403 before the bot ever reaches it. Check your security or bot-management settings.
3. Test the response directly. Simulate each bot's user-agent and see what status code you get back. A 200 means it can read the page; 403 or 503 means something is blocking it.

You can test from a terminal:

curl -A "GPTBot" -I https://yourdomain.com/
curl -A "OAI-SearchBot" -I https://yourdomain.com/
curl -A "ClaudeBot" -I https://yourdomain.com/
curl -A "PerplexityBot" -I https://yourdomain.com/

If any return 403 Forbidden or 503, a firewall or CDN rule is blocking that bot regardless of what robots.txt says. A full crawl review like our SEO audit checks all three layers at once, including server logs to confirm which bots actually reach your pages.

What's the difference between GPTBot and OAI-SearchBot?

This is the single most important distinction for AI visibility, so it's worth being precise.

• GPTBot collects data to train OpenAI's models. Whatever it reads may inform how future ChatGPT versions understand the world — but there's no direct, immediate link back to your site.
• OAI-SearchBot fetches pages in real time so ChatGPT can cite a current source when it answers a user. This is the bot tied to live citations and the clickable links users see in ChatGPT search.

The same pattern repeats across vendors: ClaudeBot trains, Claude-SearchBot fetches for live answers. PerplexityBot does both jobs for Perplexity. So if your goal is to be recommended and cited today, the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are non-negotiable. The training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) are a separate, philosophical decision about whether you want your content shaping future models.

Should I allow or block AI crawlers in robots.txt?

For most businesses that sell something, allow the search and citation bots and decide the training bots case by case. AI answer engines are becoming a real discovery channel; blocking the bots that feed live citations is the same as asking Google to deindex you. Here's a simple decision table.

Your situation	GPTBot / ClaudeBot (training)	Search bots (OAI-SearchBot, Perplexity, Claude-Search)	Google-Extended
Service business / B2B / local wanting more visibility	Allow	Allow	Allow
Publisher / media with licensing or paywall concerns	Block or license	Allow (you still want citations)	Block
SaaS / lead-gen competing on AI recommendations	Allow	Allow	Allow
Premium original research / proprietary data	Block training	Allow search	Block
You have no AI strategy yet	Allow (default)	Allow	Allow

The honest default for a growth-focused business: allow everything. You only need to start blocking when you have a concrete reason — a content-licensing deal, a paywall you're protecting, or proprietary data you don't want absorbed into a model. Blocking "just in case" usually costs more visibility than it protects. If you're weighing this against your wider plan, our SEO strategy work maps crawler policy to your actual revenue goals rather than fear.

Does blocking AI crawlers hurt my visibility in AI search?

Yes — directly and predictably for the search bots. If OAI-SearchBot can't fetch your page, ChatGPT can't cite it. If PerplexityBot is blocked, you won't appear in Perplexity's sources. Reports from Ahrefs and others suggest AI answer engines are a fast-growing slice of how people research purchases, so cutting those bots off removes you from an expanding shelf.

Blocking the training bots is more nuanced. It won't remove you from a live answer today, because live answers come from the search/fetch bots and from existing model knowledge. But over time, if your brand never appears in training data, models are less likely to "know" you exist when no live source is fetched. For most brands, being present in both layers is the safe bet.

One important caveat: blocking AI crawlers is not a privacy or copyright shield. robots.txt is a request, not a lock. Well-behaved bots honor it; not every scraper does. If you genuinely need content protected, use authentication or a paywall — not a Disallow line.

A ready-to-use robots.txt for AI crawlers

Here's a starting template that allows the major AI bots while keeping low-value paths out of the way. Swap in your real domain and adjust the disallowed paths to match your stack.

Sitemap: https://yourdomain.com/sitemap.xml

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search
Disallow: /*?*sort=
Allow: /

# Search / citation bots — allow these for AI visibility
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Training bots — allowed by default; block only with a reason
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

Two rules to live by. First, never block a page in robots.txt that you also want indexed or cited — a blocked page can't be read at all. Second, don't block CSS or JS, because most AI crawlers don't run JavaScript and need clean, server-rendered HTML to extract your facts. Pair an open robots.txt with answer-first content, real headings, lists, and tables, and schema markup, and you make it genuinely easy to be quoted.

Allowing AI crawlers is necessary but not sufficient — it gets you in the room. Being the answer they choose is the work covered in our complete guide to AI search optimization, and it's the difference between being crawlable and being recommended.

The quick audit checklist

• Open /robots.txt — no Disallow: / under AI user-agents you want to allow
• Check your CDN/WAF for an "AI bots" block toggle (often on by default)
• curl each major bot's user-agent and confirm a 200 response
• Confirm Googlebot is not blocked (that controls AI Overviews)
• CSS and JS are crawlable; critical content is server-rendered
• Sitemap is referenced and lists clean, indexable URLs

Ready to find out what AI sees on your site?

Most sites we review are blocking at least one bot they didn't mean to — usually at the firewall, not in robots.txt. If you want a clear picture of which AI crawlers can read your site and what to fix first, book a free strategy call and we'll walk through it together.