Quick answer: Major AI companies now run separate bots for separate jobs: one to gather training data, one to build a search index, and one to fetch pages when a user asks. They have different user agents so you can treat them differently. Blocking the training crawler doesn’t have to mean disappearing from AI search, and blocking everything from one company may cost you more than you meant to give up.
If you’ve looked at your server logs recently, you may have noticed that a single AI company seems to visit under several names. That’s deliberate, and it’s one of the more useful changes in how the web works, because it lets you say yes to some uses of your content and no to others.
Three jobs, three bots
The split generally follows three purposes:
Training. A crawler that collects content to train future models.
Search indexing. A crawler that builds the index an AI assistant searches when it answers questions, similar to a traditional search engine.
User-initiated fetching. An agent that retrieves a specific page because a person, in a chat, asked for it.
As of this writing, the pattern looks like this among the best-known operators:
OpenAI: GPTBot (training), OAI-SearchBot (search), ChatGPT-User (user-initiated fetches).
Anthropic: ClaudeBot (training), Claude-SearchBot (search), Claude-User (user-initiated fetches).
Perplexity: PerplexityBot (search indexing) and Perplexity-User (user-initiated fetches).
Common Crawl: CCBot, which builds a large public web archive that many AI projects train on.
These names change. Companies add bots, rename them, and revise what each one does. Before you copy any of this into a live configuration, check each operator’s own crawler documentation. They all publish one, and it’s the only authoritative source.
Google and Apple work differently
Google doesn’t send a separate training crawler. Googlebot does the fetching, and Google offers a robots.txt token called Google-Extended that controls whether content can be used for Gemini model training and some other AI products. Blocking Google-Extended doesn’t affect your ranking in Google Search.
There’s a catch that trips up many site owners. Google’s AI search features, such as AI Overviews, draw on content Googlebot crawls for Search, so Google-Extended doesn’t keep you out of those. If you want to appear in Google Search, Googlebot has to be able to read your pages, and then those features can use them.
Apple follows a similar model: Applebot crawls, and an Applebot-Extended token governs whether that content can be used to train Apple’s models.
Why the split matters now
For years, the practical choice was all or nothing: block an AI company’s crawler and lose everything, or allow it and give away everything. Separate bots make a middle position possible:
Allow search bots, so your pages can be found and cited in AI answers.
Allow user-initiated agents, because a real person is behind each request.
Restrict training crawlers, because they take the most and send nothing back.
Infrastructure providers are now pushing the rest of the industry toward this separation. Cloudflare announced in July 2026 that starting September 15 its default settings would block “mixed-use” crawlers, meaning bots that combine search, agent, and training work, on pages that carry advertising, for new customers, new sites, and free-plan sites. The stated aim is to push bot operators to split their crawlers by purpose so site owners can choose.
What a purpose-based robots.txt looks like
Here’s an example that allows search and user-initiated fetching while declining training. Treat it as a template to check against current vendor documentation, not something to paste in blind:
# Training crawlers: no
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
# Opt out of Google and Apple AI training (does not affect Search)
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Everyone else, including search and user-initiated agents
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Two WordPress notes. If you don’t have a physical robots.txt file, WordPress generates a virtual one, and some SEO plugins let you edit it from the dashboard. And if a physical file exists in your site root, it wins, so edits made in a plugin screen won’t appear.
The limits of this approach
robots.txt is a request, not a lock. Well-behaved bots from major companies honor it. Impostors and scrapers pretending to be a browser ignore it. Some operators also document that their user-initiated fetchers are handled differently from automatic crawlers, on the reasoning that a person asked for the page. Read each vendor’s policy before assuming a rule covers every bot with that company’s name.
Blocking also takes effect going forward only. It doesn’t remove anything a crawler already collected.
A sensible default for most businesses
If you sell services, earn leads from being found, or run a local business, being cited in AI answers is closer to marketing than to theft. For most of those sites, a reasonable starting position is: allow search and user agents, decide deliberately about training, and revisit the decision every few months, because the names and the rules are still moving.
If your content is the product, such as original research, data, or a paid archive, the calculation changes. That’s where pricing bots rather than simply allowing or blocking them comes in, which we’ll cover later in this series.
How Canary helps
A robots.txt change is a small edit with site-wide consequences, and the most common mistake is blocking something you didn’t mean to. Canary Site Monitor won’t write your crawler policy for you, but its 404 flood detection and error monitoring will show you when traffic patterns shift after a change, and its daily digest makes an unexpected spike hard to miss.