AI bot permissions in robots.txt: The complete guide

Your robots.txt file controls which bots can access your website. If the major AI crawlers are blocked - even accidentally - you are invisible to AI search. This guide shows you exactly how to fix that.

Key Takeaway

Every major AI provider runs separate crawlers for search and for training, and it is the search crawler that decides whether you can be cited. Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Google-Extended at minimum, and remember that the common ‘Disallow: /’ pattern blocks all of them at once.

Why this is the most important GEO fix

Of all the changes you can make to improve AI search visibility, fixing your robots.txt is the most urgent - because if AI crawlers are blocked, no other GEO work matters. A website with perfect schema markup and brilliant content is still completely invisible to AI search if the search crawlers cannot get in the door.

There is one distinction to get right before you write a single line, because getting it backwards produces the exact opposite of what you intended. Every major provider runs separate crawlers for search and for training, and they are governed by separate robots.txt rules. OpenAI crawls for training with GPTBot and for search with OAI-SearchBot. Anthropic crawls for training with ClaudeBot and for search with Claude-SearchBot. The search crawler is the one that decides whether you can appear in an AI answer. Allow the training crawler while blocking the search crawler and you get the worst of both: your content is used to train the model, and you are absent from the search results.

SearchScore’s interactive audit data (6,944 websites, July 2026) shows 6.9% of websites (roughly 1 in 14) block at least one major AI crawler. The vast majority do this accidentally - through legacy robots.txt rules written before AI search existed.

Check which AI crawlers can access your site

Enter your domain to see if OAI-SearchBot, Claude-SearchBot, PerplexityBot, Google-Extended and others are blocked.

Check

The most common accidental AI block

The most frequent culprit is a blanket disallow rule:

User-agent: *
Disallow: /

This tells every bot - including all AI crawlers - that they cannot access any page on your site. It is often added to staging or development sites and accidentally left in place, or added to “protect” a site from spam bots without realising it blocks everything.

The major AI crawler user-agents

Sorted by what each one actually decides. If you only act on one row, make it a search crawler.

Two user-agents you may still see recommended elsewhere, anthropic-ai and Claude-Web, no longer appear in Anthropic’s published crawler documentation. Leaving them in your file is harmless, but they are not what governs Claude access today.

The recommended robots.txt configuration

To allow all major AI crawlers while maintaining any existing rules for other bots:

# Search crawlers - these decide whether you can be cited
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

# User-initiated fetches - someone asked the assistant about your page
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

# Training crawlers - allow or disallow as a policy choice.
# Blocking these does NOT remove you from AI search.
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

# Your existing rules below
User-agent: *
Disallow: /wp-admin/
Disallow: /private/

Separating search retrieval from training data

This is the most common thing site owners want, and the most common thing they get backwards: allow AI search retrieval so you can be cited, while declining to have your content used for model training. It is entirely achievable, because both major providers give you a separate user-agent for each.

For OpenAI, allow OAI-SearchBot and disallow GPTBot. In OpenAI’s own words, a site opted out of OAI-SearchBot “will not be shown in ChatGPT search answers”, while disallowing GPTBot “indicates a site’s content should not be used in training generative AI foundation models”.

For Anthropic, allow Claude-SearchBot and disallow ClaudeBot. Anthropic documents ClaudeBot as collecting content that “could potentially contribute to their training”, and Claude-SearchBot as what indexes your content for search.

# Cite me, but do not train on me
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Reverse those and you get the worst of both outcomes: your content trains the models and you are absent from the answers. Check each provider’s published crawler documentation before you deploy, as these policies do change.

How to test your configuration

1. Visit yoursite.com/robots.txt and review the rules

2. Request a page with the crawler’s user-agent set and check you get a 200, for example curl -A "OAI-SearchBot" -I https://yoursite.com/. Search Console’s robots.txt report shows you the file Google fetched, but it cannot test other providers’ user-agents

3. After making changes, wait 24 to 48 hours before re-testing, as crawlers cache robots.txt files

4. Run a SearchScore audit to verify AI citability signals are now passing

Back to pillar

- Technical GEO: How to Optimise Your Website for AI Search →

S

Ronnie Huss

GEO Research & Analysis

The SearchScore editorial team researches and writes about generative engine optimisation, AI search visibility and the signals that determine whether your website gets cited by ChatGPT, Perplexity and Google AI Overviews.

Sources & Further Reading

- OpenAI – OpenAI crawlers: OAI-SearchBot, GPTBot and ChatGPT-User

- Anthropic – Does Anthropic crawl data from the web? (ClaudeBot)

- Perplexity – PerplexityBot crawler documentation

- Google Search Central – Overview of Google crawlers and user agents

Frequently Asked Questions

What are the main AI crawler user-agent names?

They split into two groups. Search crawlers, which decide whether you can be cited: OAI-SearchBot (OpenAI/ChatGPT search), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), Googlebot (Google, including the index AI Overviews cites from) and Bingbot (Microsoft Copilot). Training crawlers, which do not affect search visibility: GPTBot (OpenAI) and ClaudeBot (Anthropic). Google-Extended is a separate control for Gemini grounding and training. ChatGPT-User and Claude-User handle fetches a person triggered by asking about your page.

How do I check if my robots.txt is blocking AI crawlers?

Visit yoursite.com/robots.txt and look for any User-agent: * rule with Disallow: / - this blocks all crawlers including AI bots. Then check for explicit blocks on the search crawlers specifically: OAI-SearchBot, Claude-SearchBot and PerplexityBot. To verify one directly, request a page with that user-agent set, for example curl -A “OAI-SearchBot” -I https://yoursite.com/.

Should I allow AI crawlers for training data collection?

This is your choice, and both major providers let you make it. Allow OAI-SearchBot and Claude-SearchBot so you can be cited, and disallow GPTBot and ClaudeBot if you do not want your content used for training. Getting these the wrong way round is the common mistake: it trains the models on your content and removes you from the answers at the same time.

Check your AI visibility

Enter your URL at SearchScore for a free AI visibility Score. See how ChatGPT, Perplexity and Google AI see your site - and exactly what to fix.

See whether your fixes worked

The Tracker re-checks your site every week and shows which scores moved. Start with the free scan, then open a Founders Tracker account from your fix list. No card.

Run my free scan →