AI crawlers: what to allow in robots.txt for GPTBot, ClaudeBot and PerplexityBot
Updated: 4 August 2026
If your robots.txt blocks AI crawlers, your content never reaches the models and you're invisible in their answers. To be seen, explicitly allow the key ones: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended. It's a few lines with an immediate effect.
Which bot does what
They don't all do the same job, and the distinction matters when you decide what to allow:
- GPTBot — collects content for training OpenAI's models.
- OAI-SearchBot — feeds ChatGPT's live search (this is what affects what users see today).
- ChatGPT-User — fires when a user explicitly asks ChatGPT to open your page.
- ClaudeBot — Anthropic's equivalent for Claude.
- PerplexityBot — for Perplexity's answer engine.
- Google-Extended — not a crawler; it's a switch controlling whether your content is used in Google's AI products.
The problem you can't see
Many sites block these bots without knowing: a security plugin's default, a CDN setting, or someone copying a robots.txt off the internet.
The result is silent — no error, you simply never appear in the answers. It's worth checking before anything else.
An example robots.txt that allows AI
The logic is simple: one group per bot with explicit permission. For the essentials:
- User-agent: GPTBot / Allow: /
- User-agent: OAI-SearchBot / Allow: /
- User-agent: ChatGPT-User / Allow: /
- User-agent: ClaudeBot / Allow: /
- User-agent: PerplexityBot / Allow: /
- User-agent: Google-Extended / Allow: /
- And don't forget the Sitemap: line with the full URL of your sitemap.
Should I allow all of them?
That's a business decision, not a technical one. If you depend on customers finding you, the answer is almost always yes — blocking the bots makes you invisible exactly where search is moving.
If you have paid content or an archive you don't want used for training, you can allow the search bots (OAI-SearchBot, PerplexityBot) and restrict the training ones. That's a legitimate middle ground.
What to check now
- Open your site with /robots.txt at the end and read what's there.
- Look for any Disallow affecting the user-agents above.
- Make sure there's a Sitemap line.
- Check your CDN or firewall too — it may block bots before they ever reach robots.txt.
Frequently asked questions
If I allow the bots, are they “stealing” my content?
They use it to produce answers, often citing the source. The practical question is: would you rather be mentioned, or have your competitor mentioned? For most businesses the visibility is worth more.
Is robots.txt binding?
It's a convention that major providers honour. It isn't a technical barrier — for real blocking you need server or CDN level rules.
How fast does the change show?
Unblocking is a precondition, not a guarantee. The bots need to crawl again and the models to refresh — think weeks to months.
See where you stand in AI — free
The free micro-audit shows you in 2 minutes whether ChatGPT mentions you for your own keywords.
Start the free check →