All the online chatter about AI crawlers made me realize I have no concrete numbers, so I added more stats to this blog. Let’s look at the numbers together and marvel at all the bot activity.
Contents ↑ top ↑
The numbers
Without further ado, you can see live stats here. Below is a snapshot of last week.
My blog is about 100 pages and each of them is represented as a triplet in the visualization. Each triplet contains:
- GETs: Number of HTTP
GETrequests served. - Likely human: There is a bit of JavaScript at the top of every page that waits 5.1s after page load and then does a
POSTrequest. This stat is the number of times thePOSTrequest was received. This identifies humans, for the most part. - Read to end: There is a second bit of JS at the bottom of every page that, when scrolled into view, waits 5.1s and then does a
POSTrequest. The stat counts these requests. This checks that the reader scrolled down.
The visualization shows the total at the top. Then it shows the stats for the non-repeating pages like the front page, my bookshelf, and the contact page. Then it shows the stats for posts—these are just numbered with 73 being the most recent one. Finally, it shows stats for the various “index” pages that are just lists of posts.
Observations
The first thing to notice is that my blog is much more popular with bots than with humans. The totals are 9141/350/68. Dead Internet theory is real—only 5% of my traffic might be humans and only 1% scrolled to the bottom.
The next thing to notice is that every page has GETs, often without any corresponding human activity. It’s clear the site got crawled at least 10 times last week. This confuses me a bit because most of the content is static. For example, post 1 hasn’t changed since 2012.
My frontpage is the most popular page. Most of this is Mastodon which we’ll talk about later, but even excluding that, it’s still hundreds of requests per day. I assume bots are constantly pinging it to see if anything has changed. Most of these are SEO services like Ahrefs and Semrush. I don’t use them, so these people are just voluntarily pinging my obscure blog tens of times per day.
My bookshelf is the second most popular page with bots. In particular, it’s much more popular than my contact page which is at the same level of navigation. Francesco guesses it’s because it has lots of “content”, but I think you could flip this argument and say that it should be ignored because it looks like a link farm.
Also, I think there might be humans still clicking on posts I wrote in 2007. These used to live on a Wordpress blog and the URL changed when I imported them, so I don’t think there are any external links to them. They also don’t rank highly in search results, so I have no idea how people are finding these.
robots.txt
All the above is ostensibly without AI crawlers because I added all of ai.robots.txt to my robots.txt the day before I started collecting these stats.
It took OpenAI and Anthropic a few days to get the message, but they did back off eventually. I also added Ahrefs and Semrush halfway through the week and they also backed off.
In fact, the only bot that flagrantly ignores robots.txt is DuckAssistBot. So, shame on DuckDuckGo.
But also shame on all the bots lying about their user agents and pretending to be browsers. We’ll get to them later, but you can see lots of these every day in the live stats.
Bot sophistication
The bots are a lot more sophisticated than I thought they were. My model before embarking on this project was Internet Archive which pretty much just downloads static files. This is really obvious on captures of my front page where the dynamic blogroll content is missing.
Googlebot is a different beast altogether. I’m going to be talking exclusively about Googlebot because it’s easy to test for and it’s probably the most sophisticated one.
Here’s my thought process as I was trying to identify Googlebot through its behaviour.
“Ok, crawlers don’t execute JavaScript, so I can just identify them by doing a
GETthrough JS on page load.”
That doesn’t work. Googlebot absolutely executes JS and you can see this by searching for text from the blogroll and filtering down to just this site with the site: keyword.
“Ok, so it executes JavaScript, presumably because of React websites. But it won’t execute
POSTs because those potentially modify data. So, let’s replace the on-loadGETwith aPOST.”
It executes POST requests too. Googlebot is pretty easy to test for because you can just ask it to hit a page through Google Search Console.
“Ok, but it’s not going to scroll to the bottom of the page, trigger HTMX
revealedevents, then do aPOST, right?”
It does all of that. I assume this is to support gallery pages that lazy-load images through JavaScript instead of setting loading="lazy" on the <img> elements.
“Ok, but what if I wait a few seconds before doing the
POST? Googlebot probably has computation limits, so surely it’s not going to keep JavaScript running if the DOM isn’t being actively modified for multiple seconds. Let’s wait 5.1s before doing the on-loadPOST.”
It actually waits 5.1s and then does the POST. This really surprised me.
“But what if I combine the last two things? Will it scroll to the bottom of the page, wait 5.1s, and then do a
POST?”
No! This is the one thing that it consistently does not do.
I say consistently because there are actually several versions of Googlebot. Some of them don’t execute JS at all. Some of them do, but don’t wait and do the POST request. And some do all the things. I assume Google have a pyramid approach where they first run a cheap crawler on websites to see if anything has changed and only send the expensive crawler if they think the changes warrant it.
So, in my stats above, at least some incarnations of Googlebot and other crawlers will appear in the Likely human numbers. As far as I can tell, the only numbers that are definitely not bots are the Read to end ones.
Humans
My stats are geared towards excluding all the bots. This means they certainly identify some humans as bots.
For example, did the human load a page, but close it before 5.1s? Bot. Did they read to the end, but not wait 5.1s? Bot. Browsing with JS disabled? Also a bot.
Bots
There are a lot of bots, so let’s group them by their user-agents.
Mastodon
To address the elephant in the room, the closest thing I’ve seen to a DDOS is Mastodon. Of the 9150 GETs I saw last week, 5350 were Mastodon.
What happens is that when you post a link to Mastodon, about 500 servers will hit the URL in the next minute to download metadata. Another 500 will hit it over the course of the day.
If you have links in your Mastodon profile, you get a constant stream of servers rechecking that the pages still contain your Mastodon username. This accounts for most of the hits on my front page and bookshelf. This process is also really spiky—yesterday, the maintainer of curl replied to one of my toots and caused Mastodon servers to do some 4,000 requests to my site in a few minutes.
I’ve had an account on Mastodon for years and I didn’t do anything special last week, so this is just business as usual.
Feed readers
There were also some 300 requests from feed readers like NewsBlur, FreshRSS, and gofeed. Newsblur is the biggest/noisiest by far with around 250 requests. None of them had JavaScript enabled.
To be clear, these are requests to the posts themselves, not to the Atom feed (which I don’t track). I do include the full article contents in the feed, but I guess some people prefer to see the original pages.
SEO crawlers
I’ve already mentioned Ahrefs and Semrush, but they were just the most aggressive. There’s also BLEXBot from SE Ranking, Petalbot, MJ12Bot from Majestic, SeznamBot, Barkrowler from Babbar, ZoominfoBot, and others.
The SEO business is much more spread out than I thought. I also remain confused as to why these people are doing tens of requests every day to my obscure blog.
Search engine crawlers
There’s Googlebot and Bingbot as expected, but there were also requests from search engines I had never heard about like Ibou.
AI crawlers
There’s been a lot of hubbub this year about AI crawlers. SourceHut and Codeberg have both complained that they’re getting hammered by these bots. Drew DeVault had some characteristically strong words to say about them. Anecdotally, I have heard sysadmins from other sites complain as well.
My blog is not SourceHut or Codeberg. Notably, my server just delivers static files and has no expensive endpoints, so it can’t really get “hammered” in any meaningful sense. For me, the AI crawlers are at most an annoyance and not a problem.
The day before the week of stats, I added all of ai.robots.txt to my robots.txt. Basically all the AI crawlers that set proper user-agents respected this, albeit with a couple of days of delay. These were Claude-SearchBot, Claude-User, ChatGPT-User, OAI-SearchBot, and Amazonbot. The only one that continues to ignore robots.txt is DuckAssistBot.
That said, the stats also show a lot of suspicious user-agents. By my calculations, about 2,000 of the 9,150 GETs last week were like this.
Fake user-agents
Some bots are definitely lying about their user agents. The closest thing to a DOS I experienced was Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0 doing 350 requests in 1 minute. Firefox 47 is from 2016 and it showed up in my stats because it didn’t have JS enabled, so this is clearly fake.
Speaking of old browsers, there’s a recurring pattern of requests where each is done by a different version of a browser. Here’s the really egregious example of Firefox releases from 2012 doing it:
Also, you know how every browser UA starts with Mozilla/5.0? Well, I have the honour of getting exactly one request from 6.0:
This pattern of old browsers is very easy to spot in the live stats because there’s so many of them that they take up most of the vertical space in the list of likely bots.
I assume these are crawlers attempting to skirt user-agent based blocks, but surely it would be better to pretend to be the latest Chrome, no? Apparently, not.
Funny user-agents
There were also some funny user agents like ThinkChaos:
Mozilla/5.0 (compatible; ThinkChaos/0.3.0; +In_the_test_phase,_if_the_ThinkChaos_brings_you_trouble,_please_add_disallow._Thank_you.)Mozilla/5.0 (compatible; Thinkbot/0.5.8; +In_the_test_phase,_if_the_Thinkbot_brings_you_trouble,_please_block_its_IP_address._Thank_you.)
I’m really glad they gave me permission to block them. Otherwise, I might have agonized over the decision for months. I also can’t help but notice that they seemingly dropped support for robots.txt between versions 0.3.0 and 0.5.8.
I also got a hit from an Xbox browser. I’d say I’m flattered that somebody is reading scvalex.net in between their Call of Duty matches, but they didn’t have JS enabled, so this is probably a bot.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; Xbox; Xbox One) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edge/44.18363.8131
I’d be remiss if I didn’t mention curl/8.14.1 which did hundreds of requests. I spent days wondering who this jackass who didn’t set a proper user-agent was and why they were hitting my front page every 10 minutes like clockwork day in and day out. Then I remembered that I had set up my own health checks with curl, so I was the jackass. Well, lesson learned: don’t be like younger Alex, or like the people hitting my blog with Python/3.10 aiohttp/3.12.15, python-urllib3/2.2.3, or colly - https://github.com/gocolly/colly—always set a proper user-agent.
No user-agents
Addendum 2025-12-03: After adding some more stats, it turns out there’s yet another kind of user-agent shenanigans—missing user-agents. In this week with 4.8k GETs, there were about 400 requests with no user-agents at all.
All the HTTP libraries I know of set a user-agent by default, so these are probably bots that have explicitly been configured to not send the header at all. I’d call this clever, but it makes the requests trivial to identify once you know the trick.
Script kiddies
In addition to user-agents, I also started tracking requests that error out. As expected, it’s mostly script kiddies trying canned exploits.
Here are the ones hoping to get cloud credentials from incorrectly deployed env files:
GET /.envGET /secrets.envGET /dev/.envGET /.env.exampleGET /.env.bak
Here are ones hoping you deployed your git repo:
GET /.gitignoreGET /.git/objectsGET /.git/objectsGET /.git/HEAD
Here are ones trying to interact with Wordpress:
GET /wp-content/plugins/hellopress/wp_filemanager.phpGET /wp-login.php?action=registerGET /wp-includes/Text/wp-conflg.php
I want to highlight that there’s a typo in the last one: wp-conflg.php. I think the only thing more embarrassing than being this particular script kiddie is getting hacked by him.
There were also a few trying to interact with PHP and Java:
GET /php/php.exe?%ADd+cgi.force_redirect%3D0+%ADd+cgi.redirect_status_env%3D0+%ADd+fastcgi.impersonate%3D1+%ADd+open_basedir%3D+%ADd+disable_functions%3D+%ADd+auto_prepend_file%3Dphp://input+%ADd+allow_url_include%3D1+%ADd+allow_url_fopen%3D1GET /components/com_jbcatalog/libraries/jsupload/server/php/index.php?file=tf2rghf.jpg
All in all, there were not as many of these as I expected. The script kiddies are much better at backing off than the mystery crawlers.
Implementation
Adding stats like these to my server was fairly easy in theory, but a bit complicated in practice because I have a multi-node setup, so I also needed to write some clustering code. That will be a subject of a different post. Here, let’s see how you’d add stats tracking to a single-node axum server.
We start with the common setup of having several handlers for different kinds of routes, all sharing the same state, and falling back to the static file service if none of the special routes were matched. Our state is inside of an Arc<Mutex<_>> because we’ll need to modify the stats from different threads.
type State = Arc<Mutex<StateInner>>;
struct StateInner {
// …
stats: HashMap<String, u32>,
errors: HashMap<String, u32>,
}
async fn main() {
// …
let routes = Router::new()
.route("/robots.txt", get(get_robots_txt))
.route("/stats", get(stats_handler))
// …
.fallback_service(tower_http::services::ServeDir::new("dist_dir"))
.layer(tower_http::compression::CompressionLayer::new())
.with_state(state);
let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
.await
.unwrap();
axum::serve(listener, app).await.unwrap();
}
Since axum is built atop tower_http, we can use middleware from that crate. That’s were ServeDir and CompressionLayer come from.
Middleware in this context just means bits of code that modify requests and responses in a layered approach. For example, ServeDir is a middleware that takes a request, finds a file on disk, and returns its contents as a response. CompressionLayer is a middleware that is layered atop other middlewares like ServeDir and compresses responses. We can also write our own middleware and the easiest way is with axum::middleware:from_fn:
async fn main() {
// …
let routes = Router::new()
.fallback_service(tower_http::services::ServeDir::new("dist_dir"))
.layer(tower_http::compression::CompressionLayer::new())
.layer(middleware::from_fn(my_middleware));
// …
}
async fn my_middleware(request: Request, next: Next) -> Response {
// Modify the `Request` in some way…
// Call the next level of middleware to get a `Response`.
// We could also create our response here and ignore any inner layers.
let response = next.run(request).await;
// Modify the `Response` in some way…
// Pass the `Response` to the layer above in the middleware stack.
response
}
In the above example, the path of a Request is my_middleware → CompressionLayer → ServeDir. The path of a Response is ServeDir → CompressionLayer → my_middleware.
Let’s use this pattern to add stat tracking:
async fn main() {
// …
let routes = Router::new()
.route("/robots.txt", get(get_robots_txt))
.route("/stats", get(stats_handler))
// …
.merge(
Router::new()
.fallback_service(ServeDir::new("dist_dir"))
.layer(middleware::from_fn_with_state(state.clone(), track_stats))
)
// …
.with_state(state);
// …
}
async fn track_stats(
State(state): State<SharedState>,
request: Request,
next: Next,
) -> Response {
let uri = request.uri();
let path = uri.path().to_string();
let method = request.method().clone();
let response = next.run(request).await;
let status = response.status();
let is_html = response
.headers()
.get(CONTENT_TYPE)
.is_some_and(|x| x.as_ref().starts_with(b"text/html"));
if is_html && status.is_success() {
let entry = state.lock().await.stats.entry(path);
*entry += 1;
} else if !status.is_success() && !status.is_redirection() {
let entry = state.lock().await.errors.entry(format!("{method} {path}"");
*entry += 1;
}
response
}
If we just layered track_stats on the main Router, every request would get tracked, even the ones to the /robots.txt and /stats special handlers. We only want to apply track_stats to ServeDir, so we create a new Router with just ServeDir and track_stats, then merge it into the top router.
We also want to track stats exclusively for pages and not other assets like images, so we check that the Response has Content-Type: text/html in track_stats.
That’s all it takes. You can make this as complex as it needs to be, but this is the basic shape of the solution. For example, I have more middleware to handle things like caching (see web_server.rs). I also track stats at a more granular level, but that’s just making the data structures more complex in the obvious way.
Follow-up
To sum up, adding stats to my blog showed a surprisingly large amount of bot traffic. There were some 9,000 requests in a week. Half were Mastodon doing its best impression of a thundering herd and most of the other half were crawlers. A few of those were the big search engines as one would expect, but most of the crawler traffic came from SEO firms and people using fake user-agents.
This large amount of crawler activity perplexes me. This isn’t Wikipedia, StackOverflow, Github, or Reddit—why is my obscure blog of 100 static pages getting 5,000 crawler requests per week? I guess the answer must be that it’s cheaper to just re-crawl the Internet than to put some effort into identifying the bits of it that actually change. This reminds me of the SourceHut Go module mirror debacle from a couple of years ago where Google was git clone’ing repos in a loop because they couldn’t be bothered to do use git pull.
So, what now? The crawlers aren’t causing problems for my static site, but I don’t really like them on principle. There’s off-the-shelf solutions like Anubis and go-away, but I like to write my own stuff for my blog (that’s why it’s a custom Rust server).
The SEO crawlers seem to respect robots.txt, so I’ll just add them to it. As for the mystery crawlers, maybe if a user-agent doesn’t pass my “likely human” test a few times, I bounce all further requests from it to a captcha page until they pass it at least once.