Quick answer: Every bot that visits your site is run by someone who expects to get something from it: a search index, training data, an answer for a user, competitive intelligence, or a way in. Knowing what each one wants, and what it gives back, is the only sensible basis for deciding which to welcome, which to charge, and which to shut out.
Most advice about bots starts with a list of names to block. That skips the more useful question. Crawling the web costs money in bandwidth, computing power, and engineering time, so nobody runs a bot for fun. Every visitor in your server log is there because someone expects a return. Figure out the return and the right response usually becomes obvious.
The original bargain: search
Search engines built the first large-scale crawlers, and they came with a deal most site owners accepted without thinking about it. The crawler reads your pages, the search engine indexes them, and when someone searches for what you offer, they get sent to you.
That exchange is why blocking Googlebot is almost always a mistake. The bot costs you a little server load and pays you back in visitors, and for most businesses that’s the best trade on the web.
Training data: value that doesn’t come back
AI training crawlers collect text to teach language models how language and the world work. What the model learns from your page gets folded into its general knowledge. It doesn’t link back, cite you, or send visitors.
That’s not automatically bad. If you publish general information and want your ideas to spread, being part of what models learn may suit you fine. But it’s a one-way exchange, and it’s why training crawlers are the category most site owners now restrict first.
Retrieval and AI search: a partial bargain
AI assistants increasingly answer questions by looking things up at the moment someone asks. The crawlers that feed those lookups work more like search: they index your content so the assistant can quote or summarize it, often with a link to the source.
The bargain here is real but thinner than classic search. You may be cited, and some readers click through. But many questions are fully answered in the chat window, and the visitor never arrives. For sites that sell expertise or earn from ads, that difference is the entire business question.
User-triggered agents: one person’s errand
When someone asks an assistant to read a specific page, compare prices, or check opening hours, the assistant fetches your page right then. That isn’t crawling in the usual sense. It’s one person’s task carried out by software.
These visits matter more than their numbers suggest, because a real potential customer is behind each one. Blocking them can mean blocking the person who was about to hire you, just because they asked a machine to check you out first.
Commercial crawlers: someone else’s product
SEO platforms, price trackers, and market intelligence tools crawl constantly because their customers pay for what they find. A backlink tool maps who links to whom. A price monitor watches competitors’ product pages. A lead-generation scraper harvests contact details.
You’re the raw material for someone else’s product. Some of it is harmless and a few are tools you use yourself. The value exchange is usually zero for you, which is why many sites rate-limit this category rather than blocking it outright.
Monitors: bots you invited
Uptime checkers, feed readers, and link-preview fetchers are bots doing a job for you or for your readers. When someone pastes your URL into a chat and a neat preview card appears, a bot made that card.
It’s easy to forget this category exists until an overzealous firewall rule breaks it.
Scanners: the bots that want in
Vulnerability scanners exist because a known flaw in a popular plugin is a key that fits thousands of doors. When a plugin vulnerability is published, automated scanning for it tends to follow quickly. The scanner doesn’t care who you are. It’s checking whether you’re one of the doors it can open.
This category gives nothing back and takes whatever it can. It also ignores robots.txt, because asking politely was never the plan.
A simple way to decide
For each family, ask two questions: what does it take, and what does it give back?
Takes a little, gives a lot: search crawlers, your own monitors. Allow.
Takes a little, gives something to a real person: user-triggered agents. Usually allow.
Takes a lot, gives something indirect: AI search and retrieval. A judgment call that depends on how you earn.
Takes a lot, gives nothing back: training crawlers, most commercial scrapers. Restrict, rate-limit, or, increasingly, charge. A later post in this series covers how charging bots works.
Takes whatever it can: scanners. Block at the firewall and keep plugins patched.
The categories will keep shifting as AI companies split and merge their crawlers and new agent products appear. The questions won’t change. A bot is a business decision someone else made about your site, and you’re allowed to make one back.
How Canary helps
You can’t answer “what does it take?” without seeing the traffic, and the most expensive bot traffic is the kind that shows up as errors. Canary Site Monitor’s 404 flood detection surfaces bursts of requests for pages that don’t exist, the signature of scanners and careless crawlers, and its database and plugin checks catch the damage when heavy traffic lands at the wrong moment.