AI crawlers in robots.txt: which to allow and which to block
By Jasper Paauwe. Updated on .
Which crawler does what
The providers separate search from training, and document that themselves. If you block only the training crawlers, you can still appear in search results.
| Crawler | From | Purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | Search results in ChatGPT |
| GPTBot | OpenAI | Model training |
| Claude-SearchBot | Anthropic | Search results in Claude |
| ClaudeBot | Anthropic | Model training |
| PerplexityBot | Perplexity | Search results, not for training |
| Bingbot | Microsoft | Bing and Copilot |
| Google-Extended | Training and grounding of Gemini, not Google Search |
Crawlers that do not always follow robots.txt
Some requests do not come from a crawler but from a user who asks something. Perplexity writes that Perplexity-User generally ignores robots.txt in that case. Under the standard, robots.txt is also a request to crawlers and not a form of security. If you really want to keep something out of reach, put it behind a login.
JavaScript and the firewall
In measurements by Vercel and MERJ from December 2024, none of the major AI crawlers executed JavaScript. Text that only loads in the browser is therefore invisible to them. Look at the page source to check what is really in the HTML.
A firewall or CDN can stop crawlers that robots.txt does allow. Our check only sees robots.txt, so check those settings yourself as well.
Sources
- Overview of OpenAI Crawlers (OpenAI, 2026)
- Does Anthropic crawl data from the web? (Anthropic, 2026)
- Perplexity Crawlers (Perplexity, 2026)
- Google's common crawlers (Google, 2026)
- Webmaster Guidelines (Microsoft Bing, 2026)
- RFC 9309: Robots Exclusion Protocol (IETF, 2022)
- The rise of the AI crawler (Vercel and MERJ, 2024)