The Complete robots.txt Guide for SEO and AI

robots.txt is a plain-text file at your domain root that tells crawlers which URLs they may request. It controls crawling, not indexing — a critical distinction. To keep a page out of search results, use a noindex tag, not a Disallow line. robots.txt also lets you set per-bot rules for Googlebot and AI crawlers separately.

What is robots.txt and how does it work?

robots.txt is the oldest and simplest tool in technical SEO. It lives at one fixed location — https://yourdomain.com/robots.txt — and any compliant crawler reads it before requesting pages. It follows the Robots Exclusion Protocol, which became a formal IETF standard (RFC 9309) in 2022.

The file works on an honor system. Well-behaved crawlers like Googlebot, Bingbot, GPTBot, and ClaudeBot read and respect it. Malicious scrapers ignore it entirely. So robots.txt is a traffic-management tool for legitimate bots — not a security control. If content must be private, put it behind a login, not a Disallow line.

Each rule targets a User-agent (a named crawler, or * for all) and then allows or disallows URL paths. Crawlers match the rules under the most specific user-agent block that applies to them and ignore the rest.

What does a robots.txt file actually contain?

A robots.txt file is built from a handful of directives. Here is what each one does and how widely it is supported.

Directive	What it does	Support
User-agent	Names the crawler the following rules apply to	Universal
Disallow	Blocks crawling of a path or pattern	Universal
Allow	Permits crawling of a path inside a disallowed folder	Google, Bing, most majors
Sitemap	Points crawlers to your XML sitemap (absolute URL)	Universal
Crawl-delay	Requests seconds between requests	Bing/others — ignored by Google
* (wildcard)	Matches any sequence of characters in a path	Google, Bing, major bots
$ (end anchor)	Matches the end of a URL	Google, Bing, major bots
#	Comment line, ignored by crawlers	Universal

A few rules to keep you out of trouble:

• One file, one location. It must be at the root. A robots.txt in a subfolder does nothing.
• It is case-sensitive for paths. /Blog and /blog are different.
• Order matters within a block for some crawlers; for Google, the most specific (longest) matching rule wins, not the first.
• An empty or missing robots.txt means "crawl everything." That is usually fine.

What is the difference between Disallow and noindex?

This is the single most expensive misunderstanding in technical SEO, so read it twice. Disallow controls crawling. noindex controls indexing. They are not interchangeable, and using the wrong one creates a classic, self-inflicted bug.

Here is the trap: if you Disallow a page in robots.txt, Google never crawls it — which means Google never sees the noindex tag on it. The page can still appear in search results as a bare URL with no description ("No information is available for this page") because Google knows the URL exists from links but is forbidden from reading what is on it.

Goal	Use this	Do NOT use
Keep a page out of search results	noindex meta tag or header	Disallow
Stop crawling of low-value URLs (filters, search)	Disallow	—
Remove a page already indexed	noindex + let Google recrawl	Disallow (hides the noindex)
Block a whole staging site	Password protection	Disallow alone

The correct pattern to deindex a page is: allow crawling, add noindex, wait for a recrawl. Only after it has dropped out of the index can you safely Disallow it to save crawl budget. We dig into this and the rest of the crawl-vs-index relationship in our SEO audit, because a single wrong line here can keep money pages out of search for months.

Which AI crawlers should I name in robots.txt?

robots.txt is now how you set policy for AI crawlers too, and they are split into two distinct jobs. Training bots gather data to build models. Search/citation bots fetch live pages so an AI assistant can cite a current source when it answers a user. The search bots are the ones that send you visibility and get your brand named in an answer.

Crawler	Operator	Job	Block it and...
GPTBot	OpenAI	Trains ChatGPT models	Your content won't shape future models
OAI-SearchBot	OpenAI	Fetches for ChatGPT search citations	You can't be cited in ChatGPT live answers
ClaudeBot	Anthropic	Trains Claude models	Your content won't inform Claude training
Claude-SearchBot	Anthropic	Fetches for Claude live answers	You can't be cited by Claude
PerplexityBot	Perplexity	Indexes for Perplexity answers	You vanish from Perplexity citations
Google-Extended	Google	Gemini model training only	No effect on Google Search or AI Overviews
CCBot	Common Crawl	Open dataset reused by many AI labs	You leave a widely-used training corpus

The mistake almost everyone makes: Google-Extended controls Gemini training, not Google Search or AI Overviews. AI Overviews are served by normal Googlebot. Block Google-Extended and you stay in AI Overviews; block Googlebot and you fall out of Google entirely. For a deeper walkthrough of testing each bot, see our companion piece on whether your site is blocking AI crawlers, and the strategy side in our AI search optimization guide.

How do I write per-bot rules without breaking things?

The key behavior to understand: a crawler obeys only the single most specific User-agent block that names it, and ignores User-agent: * entirely once a named block matches. This catches people out constantly.

If you write a User-agent: * block with general rules, then add a User-agent: GPTBot block, GPTBot will follow only its own block — your global Disallow lines for /cart/ and /search will not apply to it. You have to repeat shared rules inside each named block, or accept that named bots get the rules you give them and nothing more.

Follow these steps to build a clean file:

1. Start with User-agent: * and your site-wide rules — block low-value paths (cart, checkout, internal search, faceted-filter URLs), allow everything else.
2. Add a Sitemap: line with the absolute URL. This belongs anywhere in the file and applies globally.
3. Add named blocks only when a bot needs different rules — for example, to explicitly allow AI search bots or block a specific training bot.
4. Repeat any shared blocks (like Disallow: /cart/) inside each named block, since named bots ignore the * block.
5. Never block CSS or JS. Googlebot needs them to render; most AI crawlers don't run JavaScript and need clean server-rendered HTML.
6. Test before you ship in Google Search Console's robots.txt report and by curling each bot's user-agent.

What does a good robots.txt look like in 2026?

Here is a sensible default for a growth-focused business that wants strong search rankings and AI visibility. Swap in your domain and adjust the disallowed paths to your stack.

Sitemap: https://yourdomain.com/sitemap.xml

# All crawlers: block low-value paths, allow the rest
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search
Disallow: /*?*sort=
Disallow: /*?*filter=
Allow: /

# AI search / citation bots — allow for AI visibility
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Training bots — allowed by default; block only with a reason
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

The honest default for most businesses is allow everything and only block when you have a concrete reason — a content-licensing deal, a paywall, or proprietary research. Blocking "just in case" usually costs more visibility than it protects. For a managed setup, our done-for-you SEO service configures and monitors this alongside your wider crawl health.

How do I test and validate my robots.txt?

Never deploy a robots.txt change blind — one stray Disallow: / can deindex an entire site. Validate in three ways:

• Google Search Console has a robots.txt report that shows the fetched file, flags syntax errors, and lets you test specific URLs against your rules.
• Direct fetch. Open https://yourdomain.com/robots.txt in a browser to confirm it serves the file you expect (not a 404 or an HTML error page).
• Simulate each bot from a terminal to confirm it gets a 200, not a 403 from your CDN or firewall:

curl -A "Googlebot" -I https://yourdomain.com/
curl -A "GPTBot" -I https://yourdomain.com/
curl -A "PerplexityBot" -I https://yourdomain.com/

That third check matters more than people realize. Many CDNs and firewalls (Cloudflare, Fastly, managed hosts) ship "block AI bots" toggles that are sometimes on by default. Your robots.txt can say "allow" while the firewall returns a 403 before the bot reaches it. robots.txt is only the first of three layers.

Common robots.txt mistakes to avoid

A quick reference of the errors we see most often in audits:

• Disallow: / left over from a staging build. This blocks the entire site. Always check after a launch or migration.
• Using Disallow to deindex a page — it hides the very noindex tag you need Google to read.
• Blocking CSS/JS folders, breaking rendering for Googlebot and AI crawlers.
• Assuming named bots inherit User-agent: * rules — they don't.
• Relying on Crawl-delay for Google — Google ignores it; manage crawl rate in Search Console instead.
• Treating robots.txt as security — it is a public file that lists exactly what you want hidden.
• Forgetting the Sitemap: line, missing an easy crawl-efficiency win.

Getting robots.txt right is foundational but invisible — when it works, nothing happens, and that is the point. Pair it with answer-first content, real headings, lists, tables, and schema, and you make your site both crawlable and quotable, which is the whole job covered in our broader SEO strategy work.

Ready to make sure crawlers see the right pages?

Most sites we review have at least one robots.txt or crawl-directive issue quietly costing them visibility — usually a Disallow doing a noindex's job, or a firewall blocking a bot the robots.txt allows. If you want a clear, prioritized picture of what to fix, book a free strategy call and we'll review your file and crawl setup together.