Short answer: it depends on what you want from AI. The crawlers aren't one thing. Some power AI search and answers, which can send customers to you. Others collect content to train AI models. Blocking everything labelled "AI" is a blunt move that can remove you from the results you wanted to appear in.
First, a plain-English primer
A crawler is an automated program that visits web pages. Your website's robots.txt file, at yourdomain.com/robots.txt, tells well-behaved crawlers which parts of your site they may visit.
Each vendor publishes its own crawler names, and several run more than one for different jobs. Here's what each documents.
| Vendor | Crawler | Documented purpose | What the vendor says about blocking |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surface sites in ChatGPT's search features | Sites opted out "will not be shown in ChatGPT search answers" (they can still appear as navigational links) |
| OpenAI | GPTBot | Content that may be used to train generative AI models | Disallowing it means content shouldn't be used for training; it doesn't affect search visibility |
| OpenAI | ChatGPT-User | Visits pages when a user asks ChatGPT something that needs a page | Not used to decide whether content appears in search; robots.txt rules may not apply because the user initiated it |
| Anthropic | ClaudeBot | Collects web content that could contribute to model training | Blocking excludes your future materials from training datasets |
| Anthropic | Claude-SearchBot | Indexes content to improve search result quality | Blocking may reduce visibility and accuracy in user search results |
| Anthropic | Claude-User | Retrieves pages when a Claude user asks a question that needs them | Blocking may reduce visibility in user-directed search |
| Perplexity | PerplexityBot | Surface and link websites in Perplexity search results | Allowing it is the vendor's recommendation for search visibility; not used for AI model training |
| Perplexity | Perplexity-User | Visits a page when a user asks a question | The vendor says it generally ignores robots.txt, since a user requested the fetch |
Google-Extended | A product token controlling whether crawled content may be used for training certain Gemini models and for grounding | "Does not impact a site's inclusion in Google Search nor is it used as a ranking signal" |
Two points stand out. First, search and training are separate switches at OpenAI, Anthropic and Perplexity, so you can allow one and block the other. Second, Google-Extended is only a control token: Google says it has no separate user agent string, and blocking it doesn't touch your Google Search presence, which is governed by Googlebot.
Who might choose what
This isn't one right answer. These are the trade-offs by type of business.
A local service business that wants enquiries (a plumber, a clinic, an agency). Being found is the point. Allowing the search and retrieval crawlers is the usual direction. Whether you also allow training crawlers is a judgement about how you feel about your content being used to train models; it won't change whether customers can find you in the vendors' search features, according to their documentation.
An online shop. Your product pages and prices are public by design, and being shown when someone asks an AI assistant for a recommendation is useful. The same reasoning applies: search and retrieval allowed, training a separate choice. Check that product feeds and pricing pages aren't blocked by accident.
A publisher or content-heavy business whose articles are the product. You may reasonably want to block training crawlers while allowing search ones, so you can still be cited and linked. Weigh it against how much of your traffic you expect to come from AI answers; nobody can promise you that.
A membership, paywalled or confidential site. Don't rely on robots.txt at all for protection (see below). Use access controls.
Example configurations
A robots.txt that allows AI search and retrieval but opts out of training might look like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
A robots.txt that allows everything is simply a file with User-agent: * and Allow: /, or no file at all. Have whoever manages your site test the final file before relying on it.
Four limits to keep in mind
1. robots.txt is a request, not a lock. Google's documentation says robots.txt "cannot enforce crawler behavior to your site; it's up to the crawler to obey them," and that it isn't a way to keep information secure. For private content, use passwords or access controls.
2. Some fetchers don't follow it. Perplexity says its user-initiated fetcher generally ignores robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User. If you need to control those, the vendors point to other measures such as firewall rules.
3. Changes aren't instant. OpenAI and Perplexity each say it can take about 24 hours for their systems to reflect a robots.txt update. Google says recrawling can take longer.
4. Blocking by IP address isn't a reliable opt-out. Anthropic says blocking IP addresses "may not work correctly or persistently guarantee an opt-out," and recommends the robots.txt route.
Also note that security tools, content delivery networks and hosting settings can block automated traffic independently of your robots.txt. If you've allowed a bot in your file but think it isn't getting through, ask your provider.
How to check your own site in ten minutes
- Open
yourdomain.com/robots.txtin your browser. - Look for the crawler names in the table above. Note which are allowed, which are disallowed, and which aren't mentioned at all.
- Check for a blanket
Disallow: /underUser-agent: *, which blocks everything. - Decide, per the business types above, which side of search versus training you want.
- Ask whoever manages the site to make the change and confirm it by loading the file again.
For context, our own robots.txt currently allows all of the crawlers listed in this article. That's a choice we made for our own site, not a recommendation for yours.
What this doesn't tell you
Allowing a search crawler makes you eligible; it doesn't guarantee that you'll be cited or shown. And a policy that suits you today may need revisiting as vendors change their crawlers and documentation. Treat this as a decision to review, not a one-off setting. Google's own requirements for appearing in its AI features are covered in What Google Actually Requires to Appear in AI Overviews and AI Mode. For the bigger picture on how AI search works with traditional SEO, see What Is Answer Engine Optimization (AEO)? and Why Your Business Needs to Show Up in AI Search.
Common questions
If I block GPTBot, will I disappear from ChatGPT?
OpenAI says no. It states that disallowing GPTBot doesn't affect search visibility, which is governed by OAI-SearchBot.
Does blocking Google-Extended hurt my Google rankings?
Google says it does not impact inclusion in Google Search and isn't a ranking signal.
Can robots.txt protect private pages?
No. Google says it is not a security mechanism and compliance is voluntary. Use passwords or access controls.
Which should a small business allow?
There's no universal answer. If being found through AI answers matters to you, allow the search and retrieval crawlers. Training access is a separate choice about how your content is used.
How long until a change takes effect?
OpenAI and Perplexity each say about 24 hours for their systems. Others may take longer.
Velocity Branding builds SEO and AI search visibility for small businesses and startups across the US and South Africa.
Not sure what your robots.txt is allowing right now? Book a free strategy call and we'll check it and explain what it means for your business.