Lees deze gids in het Nederlands →
AI companies run different crawlers for different jobs — training, search and user-initiated fetching — and robots.txt lets you allow or block each one separately: allow the search bots if you want citations, decide deliberately about the training bots.
- Three roles per company: training (GPTBot, ClaudeBot), search (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-fetch (ChatGPT-User, Claude-User).
- Since OpenAI’s December 2025 documentation change, ChatGPT-User may not honour robots.txt — it acts as a proxy for a human request.
- Google-Extended controls Gemini training and grounding in Gemini Apps and Vertex AI; Google states it is not a ranking signal and does not affect Search inclusion.
- Blocking all AI bots also blocks AI search: you disappear from the answers, not just from training data.
- robots.txt is a request, not a lock — documented crawlers comply, unknown scrapers may not.
AI companies use different crawlers for different jobs: some collect training data, some fetch pages for AI search answers, some act on behalf of a user. You can allow or block each one separately in robots.txt.
The smart default for most sites: allow the search crawlers (so you can be cited in AI answers), and decide deliberately about the training crawlers. Blocking everything also blocks you from AI search.
The main AI crawlers (September 2026)
- OpenAI: GPTBot collects training data. OAI-SearchBot surfaces sites in ChatGPT search. ChatGPT-User fetches pages when a user asks — and since OpenAI’s December 2025 documentation change, robots.txt rules may not apply to it.
- Anthropic: ClaudeBot for training, Claude-SearchBot for search, Claude-User for user-initiated fetches.
- Perplexity: PerplexityBot for its search index, Perplexity-User for user requests.
- Google: Google-Extended (a control token, not a separate crawler) for Gemini training and grounding in Gemini Apps and Vertex AI.
- Apple: Applebot-Extended controls use of content for Apple’s AI training.
Usually allow
Why: they decide whether you can be found and cited in AI answers.
Your call
Why: blocking them keeps your content out of future model training, with less direct effect on visibility.
| Crawler | Company | Role | Blocking costs you |
|---|---|---|---|
| GPTBot | OpenAI | Training | Little visibility; keeps data out of models |
| OAI-SearchBot | OpenAI | ChatGPT search | ChatGPT citations |
| ClaudeBot / Claude-SearchBot | Anthropic | Training / search | Same split as OpenAI |
| PerplexityBot | Perplexity | Search index | Perplexity citations |
| Google-Extended | Gemini training token | Nothing in Search | |
| Applebot-Extended | Apple | AI training control | Nothing in Search |
Google-Extended explained
Google-Extended controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google states it does not affect your inclusion in Google Search and isn’t a ranking signal. AI Overviews are part of Search and rely on Googlebot, so blocking Google-Extended doesn’t take you out of them — see what is Googlebot. Since 31 August 2026 there is a separate switch for that: the Search generative AI control in Search Console lets you leave AI Overviews and AI Mode while staying in normal results — see AI Overviews in Search Console.
Example robots.txt rules
Allow AI search, block AI training:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot are allowed by default when you don’t list them. Test any change in Search Console’s robots.txt report, and remember that robots.txt is a request, not a lock — see blocked by robots.txt.
IndexUrePoint allows all AI crawlers: we want to be cited. If you publish original work that’s your product — research, paid content — blocking training crawlers can make sense. Blocking AI search crawlers almost never does.
What about llms.txt? It doesn’t allow or block anything — it only suggests which pages matter to AI agents. See what is llms.txt for who actually reads it.
Real data from this site. Our host’s AI crawler log for 24 hours on 25 September 2026 showed 21 requests from known AI crawlers — 17 from GoogleOther and 4 from OAI-SearchBot — and none blocked. Testing with faked crawler user agents gave a very different (and misleading) picture; we explain why in our 403 guide.
Check before you publish. Our free robots.txt tester shows in one click whether GPTBot, ClaudeBot, PerplexityBot and Googlebot are allowed or blocked by your rules.
FAQ
Does blocking GPTBot remove me from ChatGPT search?
Not by itself. ChatGPT search uses OAI-SearchBot, which is controlled separately from GPTBot, OpenAI’s training crawler.
Does blocking Google-Extended hurt my Google rankings?
No. Google states Google-Extended does not affect inclusion in Search and is not a ranking signal.
Do all AI bots respect robots.txt?
The major companies’ documented crawlers do. User-initiated agents such as ChatGPT-User may not follow robots.txt in the same way, and unknown scrapers can ignore it entirely.
How do I see which AI bots visit my site?
Check your server logs or your CDN’s bot analytics for the user agents listed above.
How do I write a robots.txt rule for one AI bot?
Add a User-agent line with the bot’s exact name followed by Disallow: / to block it site-wide, or Disallow: /private/ for one section. List each bot separately.
Do AI crawlers visit small sites at all?
Yes. Server logs on modest sites routinely show GPTBot, ClaudeBot and PerplexityBot; volume scales with how often your content is referenced.