Quick answer: Most overnight traffic on a typical WordPress site isn’t people. It’s search crawlers, AI crawlers, uptime monitors, SEO tools, link-preview fetchers, and scanners probing for weak spots. Most are harmless and some are useful, but your analytics probably can’t see any of them, which is why their side effects look like mysteries.
Open your analytics dashboard at breakfast and the night looks quiet. A handful of visits, maybe a spike from a newsletter. Then open the raw server access log for the same hours and it reads like a different website: thousands of requests, steady and patient, many of them for pages that don’t exist.
Both are telling the truth. They’re just counting different visitors.
Why your analytics can’t see them
Most analytics tools work by running a JavaScript snippet in the visitor’s browser. Crawlers generally don’t run that snippet, and the ones that do are usually filtered out as known bots. So the dashboard shows you humans (roughly), and the server log shows you everything that actually asked your server for something.
That gap matters because bots don’t just read. Every request that reaches WordPress loads PHP, may query the database, and can trigger WP-Cron, which runs scheduled tasks when a page loads. On a small site, a busy crawler can be the biggest source of work your server does all night.
The main families
It helps to sort bots by what they want rather than by name, because names change and purposes don’t.
Search engine crawlers. Googlebot, Bingbot, and their peers fetch pages to build a search index. This is the oldest bargain on the web: they read your pages, and in exchange they send you visitors.
AI training crawlers. These collect text to train language models. They read broadly and repeatedly, and they don’t send visitors back.
AI search and retrieval crawlers. These build an index that an AI assistant draws on to answer questions, sometimes with a citation linking back to you.
User-triggered agents. When someone asks an AI assistant to “check this page” or “compare these three plugins,” the assistant fetches your page right then, on that person’s behalf. This is one visit for one human’s task, just made by software.
SEO and marketing tools. Backlink indexes, rank trackers, and site auditors crawl constantly so their customers can study your site.
Monitors and previewers. Uptime checkers, feed readers, and the fetchers that build link previews when someone shares your URL in a chat app.
Vulnerability scanners. Automated tools that request known weak spots: exposed configuration backups, .env files, old installer scripts, and paths belonging to plugins with published vulnerabilities. Some are run by security researchers. Many are not.
Reading the pattern instead of the name
Every one of these sends a user-agent string saying what it is, and any of them can lie. A later post in this series covers verification properly. For now, behavior is the more honest signal:
Search crawlers spread their requests out, respect your robots.txt, and mostly fetch pages that exist.
Training crawlers tend to sweep: deep archive pages, tag pages, paginated lists, everything.
User-triggered agents arrive in small, specific bursts, often one page at a time, often at hours that match their user’s day rather than a crawl schedule.
Scanners hit URLs you never had. That’s the giveaway. A request for /wp-content/plugins/some-plugin-you-never-installed/readme.txt isn’t someone lost. It’s someone checking whether you’re running a version with a known hole.
That last pattern is why a 404 log is one of the most informative files on a WordPress site. Legitimate crawlers mostly request what you’ve linked to. Scanners request what they hope you forgot.
Where it actually hurts
Most bot traffic is harmless, and some you’d be foolish to block. The trouble comes from volume and timing.
Load. An aggressive crawler on shared hosting can push a site into resource limits, and the humans who show up at 7 a.m. get a slow site, or a host’s “resource limit reached” page.
404 floods. A scanner working through a list of a few thousand paths produces a few thousand 404s, each one a full WordPress page load. If a redirect plugin logs every 404 to the database, the flood also becomes a pile of database writes.
Overnight side effects. Scheduled tasks, update routines, and background jobs all tend to run when traffic is low, which is exactly when bot traffic is the only traffic. We traced one client site’s nightly problem, every plugin deactivating itself sometime between 3 and 6 a.m., to a background task running into a database mismatch. Bots didn’t cause it, but it’s the same lesson: the hours nobody is watching are when a site does the most on its own.
What to do with this
You don’t need to become a log analyst. You need three habits:
Look at real traffic once. Pull a night of raw access logs from your host’s control panel and skim it. The first time is always surprising, and it recalibrates what “quiet” means.
Decide what you want, by family. Search crawlers: almost certainly yes. AI training: your call, and later posts in this series cover the trade-offs. Scanners: never, and they won’t ask.
Watch the side effects, not just the visitors. A bot you’ve never heard of doesn’t matter much. A thousand 404s an hour, a deactivated plugin, or a database table that’s suddenly throwing errors does.
How Canary helps
Canary Site Monitor watches the side effects. Its 404 flood detection shows when a burst of requests for pages that don’t exist is hitting your site, and where they’re coming from, so a scan looks like a scan rather than a mystery slowdown. The plugin watchdog and database health checks catch the overnight damage that tends to happen while nobody is looking, and the morning digest tells you about it before your visitors do.