AI crawlers are bots that fetch web pages for AI companies, either to train models, to power AI search answers, or to read a page because a user asked a chatbot to. You manage most of them in robots.txt with a separate user-agent token per purpose, so you can block training while staying visible in search, but robots.txt is a request, not access control.
Key takeaways
- Google-Extended controls whether Google may use your content for Gemini training and grounding; Google says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” (Google crawler docs, updated Oct 15, 2026).
- AI Overviews and AI Mode are part of Google Search: they are controlled by Googlebot rules and snippet controls such as
nosnippetandmax-snippet, not by Google-Extended. - OpenAI, Anthropic and Perplexity each use separate tokens for training, search and user-requested fetches, so “block AI” can mean three different things.
- In the 2025 Web Almanac crawl, gptbot was named in 4.5% of desktop robots.txt files (up from 2.9% in 2024) and claudebot in 3.6% (up from 1.9%).
- User-triggered fetchers such as ChatGPT-User, Perplexity-User and Google’s user-triggered fetchers may not follow robots.txt, according to their own documentation.
What counts as an AI crawler?
“AI crawler” is a loose label. When you read each operator’s documentation, the bots fall into three groups that do different jobs and need different decisions.
- Training crawlers collect pages that may be used to train AI models. Disallowing GPTBot, OpenAI says, “indicates a site’s content should not be used in training generative AI foundation models.”
- AI search crawlers index pages so an AI product can show and link to them in answers. Blocking them affects your visibility in that product.
- User-triggered fetchers load a single page because a person asked an assistant to read it, summarise it or act on it. They are not crawling the web on their own schedule.
This split matters because many sites block “AI” with one broad rule and accidentally remove themselves from AI search products that send referral traffic, while doing nothing about the fetchers that ignore robots.txt anyway.
Which user-agent tokens do the main AI companies use?
The table below lists the tokens each operator documents today. Several are new in 2025–2026, so check the source pages when you update your file.
| Operator | Token | What it does | If you disallow it |
|---|---|---|---|
| Google-Extended | Control token for Gemini model training and grounding in Gemini Apps and Vertex AI; no separate HTTP user agent | Content not used for those purposes; no effect on Google Search inclusion or ranking | |
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode | Pages drop out of Google Search entirely; use snippet controls instead to limit AI features | |
| Google-CloudVertexBot | Crawls requested by site owners building Vertex AI Agents | No effect on Google Search or other products | |
| OpenAI | GPTBot | Collects content that may be used to train foundation models | Signals content should not be used for training |
| OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT search | Not shown in ChatGPT search answers (may still appear as navigational links) |
| OpenAI | ChatGPT-User | User-initiated actions in ChatGPT | OpenAI says robots.txt rules “may not apply” |
| Anthropic | ClaudeBot | Training data collection | Future materials excluded from training datasets |
| Anthropic | Claude-SearchBot | Improves search result quality | May reduce visibility in user search results |
| Anthropic | Claude-User | Fetches pages for user queries | Claude can’t retrieve your content for users |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity search; not for training foundation models | Not surfaced in Perplexity search |
| Apple | Applebot-Extended | Control token for training Apple’s foundation models | Not used for training; still discoverable in Spotlight, Siri and Safari |
| Common Crawl | CCBot | Crawls the web for Common Crawl’s open dataset | Excluded from future Common Crawl crawls |
Two details are easy to miss. First, Apple says that if your robots.txt doesn’t mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. Second, Microsoft does not use a separate AI token: Bing’s 2023 guidance says the noarchive and nocache meta tags control how content is used in its chat answers, and pages with either tag still appear in Bing search results.
Does blocking Google-Extended remove you from AI Overviews?
No. Google’s “AI features and your website” page says AI is built into Search and that “robots.txt directives for Googlebot is the control” for crawling. To limit what appears from your pages, Google points to nosnippet, data-nosnippet, max-snippet or noindex. Google-Extended is mentioned separately, for “AI training and grounding in some of Google’s other systems.”
Google’s robots meta tag documentation (updated March 24, 2026) confirms that nosnippet “will also prevent the content from being used as a direct input for AI Overviews and AI Mode,” and that max-snippet limits how much can be used. The trade-off is that the same controls shrink your normal search snippets too. There is no Google control that removes a page from AI Overviews while leaving its regular snippet untouched.
If AI Overviews are part of your traffic picture, our guide to AI Overviews and SEO covers what Google says about eligibility, which needs nothing beyond being indexed and snippet-eligible.
How many sites already name AI bots in robots.txt?
The HTTP Archive Web Almanac 2025 SEO chapter (published January 15, 2026) counted how often each user agent is named in robots.txt files. Naming a bot isn’t the same as blocking it, because a rule can allow, disallow or set limits, but it shows which crawlers site owners care enough to address.
Year on year, gptbot rose from 2.9% to 4.5% of desktop sites (about 55%), ccbot from 2.7% to 3.5%, and claudebot from 1.9% to 3.6%. For comparison, the same chapter notes that bingbot appears in fewer than 3% of robots.txt files. Most sites still have no AI-specific rules at all.
Traffic data shows why the question has become commercial. Cloudflare’s 2025 Year in Review (data from January 1 to December 2, 2025) found AI bots averaged 4.2% of HTML requests across its network, and GPTBot made up about 7.5% of verified bot traffic. Cloudflare also reported crawl-to-refer ratios: Anthropic’s reached as much as 500,000 crawls per referral early in the year, and OpenAI’s as much as 3,700:1 in March, while Google’s ranged from about 3:1 to 30:1. Training crawls send little traffic back.
What does robots.txt not do?
RFC 9309, the Robots Exclusion Protocol standard published in September 2022, is blunt: the rules “are not a form of access authorization,” and crawlers are only “requested to honor” them. Listing private paths in robots.txt also exposes them publicly.
robots.txt works for
- Telling well-behaved crawlers what you prefer, per token
- Separating training from search for operators that document it
- Reducing crawl load from bots that respect it
robots.txt doesn’t cover
- Bots that ignore it or spoof a user agent (Common Crawl warns of fakes)
- User-triggered fetchers that may not apply it
- Content already collected before you added the rule
- Real access control: use authentication or firewall rules
Newer preference signals are emerging but aren’t standards yet. Cloudflare’s Content Signals Policy (September 24, 2025) adds lines such as Content-Signal: search=yes, ai-train=no to robots.txt; Cloudflare itself says signals “express preferences; they are not technical countermeasures.” The IETF AI Preferences working group has drafts for a shared vocabulary and for attaching it to robots.txt and HTTP headers, but they are still Internet-Drafts. And llms.txt is not a control at all: Google’s AI features page says you don’t need “new machine readable files, AI text files, or markup.”
A sample robots.txt that blocks training but keeps search
This is a common starting point for a service business that wants AI search visibility but prefers not to supply training data. Adapt it to your own decision; don’t copy it blindly.
# Google Search, including AI Overviews and AI Mode
User-agent: Googlebot
Allow: /
# Training and grounding opt-outs
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# AI search crawlers stay allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Sitemap: https://www.example.com/sitemap_index.xmlGrouping several User-agent lines above one rule is valid under RFC 9309. OpenAI and Perplexity say changes can take about 24 hours to apply, and the RFC says crawlers shouldn’t use a cached robots.txt for more than 24 hours.
How to set your AI crawler policy in seven steps
- Decide per purpose, not per company. Write down whether you want to allow training, AI search and user-requested fetches. Most marketing sites want AI search visibility; publishers and paid-content sites often refuse training.
- Read your current robots.txt. Look for a blanket
User-agent: *disallow, tokens an operator no longer lists, and rules a plugin or CDN added for you. - Check your CDN or host. Cloudflare began asking new domains whether to allow AI crawlers at sign-up from July 1, 2025, and its managed robots.txt can add rules you didn’t write.
- Write explicit groups per token. Use the tokens from each operator’s own page, keep Googlebot unrestricted unless you mean to leave Google Search, and test the file in Search Console’s robots.txt report.
- Use snippet controls for Google AI features. Apply
nosnippetordata-nosnippetonly to the sections you truly need to keep out of AI Overviews and AI Mode. - Verify in your logs. Confirm the bots you allowed are fetching and the ones you blocked have stopped. Our log file analysis guide shows how to verify real Googlebot requests with DNS lookups; OpenAI and Common Crawl publish IP lists for the same purpose.
- Review twice a year. Tokens change: Google now lists Google-Agent and a Gemini Notebook fetcher that replaces Google-NotebookLM, and OpenAI now lists OAI-AdsBot. Recheck the operator pages and update the file.
If crawl load rather than content use is your concern, start with our guide to XML sitemaps and crawl budget.
Frequently asked questions
Will blocking GPTBot remove my site from ChatGPT search?
No. OpenAI treats GPTBot (training) and OAI-SearchBot (ChatGPT search) as independent settings, so you can disallow GPTBot and still allow OAI-SearchBot. Blocking OAI-SearchBot is what removes your pages from ChatGPT search answers.
Does Google-Extended affect my Google rankings?
No. Google states that Google-Extended does not affect inclusion in Google Search and is not used as a ranking signal. It only controls use of your content for Gemini training and grounding outside Search.
Can I stop AI Overviews using my content?
Only with the same controls that limit normal snippets: nosnippet, data-nosnippet, max-snippet or noindex. Google’s robots meta documentation says these also limit use as a direct input for AI Overviews and AI Mode, so apply them selectively.
Is llms.txt worth adding?
It isn’t a crawler control. Google says you don’t need new machine-readable or AI text files to appear in its AI features, and the Web Almanac 2025 notes that no Google service currently uses llms.txt.
Sources
- Google’s common crawlers — Google for Developers, last updated Oct 15, 2026 — Google-Extended, Google-CloudVertexBot
- Google user-triggered fetchers — Google for Developers, last updated Aug 19, 2026 — Google-Agent, robots.txt behaviour
- AI features and your website — Google Search Central, last updated Dec 10, 2025 — AI Overviews/AI Mode controls and eligibility
- Robots meta tag, data-nosnippet, and X-Robots-Tag specifications — Google Search Central, last updated Mar 24, 2026 — nosnippet and max-snippet for AI features
- Overview of OpenAI Crawlers — OpenAI, read Oct 23, 2026 — GPTBot, OAI-SearchBot, ChatGPT-User
- Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic Help Center, Apr 7, 2026 — ClaudeBot, Claude-User, Claude-SearchBot
- Perplexity Crawlers — Perplexity, read Oct 23, 2026 — PerplexityBot, Perplexity-User
- About Applebot — Apple Support, Sep 4, 2026 — Applebot-Extended
- CCBot — Common Crawl, read Oct 23, 2026 — CCBot token and spoofing warning
- Announcing new options for webmasters to control usage of their content in Bing Chat — Microsoft Bing, Sep 22, 2023 — noarchive and nocache
- RFC 9309: Robots Exclusion Protocol — IETF, Sep 2022 — not access authorization; caching
- Web Almanac 2025: SEO chapter — HTTP Archive, published Jan 15, 2026 — AI user agents named in robots.txt
- Cloudflare Radar 2025 Year in Review — Cloudflare, Dec 15, 2025 — AI bot traffic share and crawl-to-refer ratios
- Giving users choice with Cloudflare’s new Content Signals Policy — Cloudflare, Sep 24, 2025 — Content-Signal lines
- AI Preferences (aipref) working group — IETF Datatracker, read Oct 23, 2026 — draft status
Cite this article: Bright Harbour Media. “AI Crawlers and robots.txt: What to Allow and Block.” Bright Harbour Media Insights, October 2026. https://brightharbourmedia.com/ai-crawlers-robots-txt/
You are welcome to reference the charts and frameworks in this article with a link back to this page.
Not sure what your robots.txt, CDN and plugins are telling AI crawlers? Our technical SEO service reviews your crawler rules, logs and snippet controls against your goals. Get in touch to request an audit.