Blocking the SemrushBot on EVILEG

I'll start sharing information about bots with SemrushBot.

SemrushBot is the official web crawler from SEMrush, one of the leading services for SEO, marketing, and competitor analysis.

I remember constantly checking backlinks and other site metrics when I first started with EVILEG. A lot of time was spent on strange SEO optimizations. Now, I honestly don't care. I feel that working with social networks and the ability to share content is more relevant now, so word-of-mouth works. Search engine optimization, while important, is not as crucial. However, SEO specialists may argue with me. That's fine. They probably know better than I do, and I'm no longer interested.

Now let's talk directly about the bot:

Bot Characteristics

User-Agent:

Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)

Purpose:

• Scans sites to collect data on SEO, traffic, backlinks, and content.
• Used by SEMrush for competitive analysis and generating reports for its clients.

Behavior:

• Operates like a regular HTTP bot, does not execute JavaScript.
• Can create significant load with aggressive scanning.
• Usually respects rules from robots.txt.

What to do with SemrushBot?

Allow

• If you don't experience load issues, you can allow access, especially if you want your site indexed for SEO analytics.

Restrict or Block

If the bot causes excessive load or unwanted scanning:

Add to robots.txt:

User-agent: SemrushBot
Disallow: /

Or block via Nginx:

if ($http_user_agent ~* "SemrushBot") {
    return 403;
}

My Choice

I decided to block it.

• I don't use SEMRush services
• SEO at this level for EVILEG is not interesting to me
• There's no place for useless bots on EVILEG.