Blocking AI bots at the Nginx level. Why and how?

Today, I'll share a small list of blocked AI bots that were scanning site content to train their neural networks based on other people's content.

But first, I'll share some of my thoughts, specifically my attitude towards the whole situation with AI assistants, neural networks, etc.

Overall, I have a respectful attitude towards such projects, particularly OpenAI with ChatGPT. It's really cool what they've created. Personally, it makes my life much easier at the moment. I now have the opportunity to make quite high-quality translations of material on the site. It's also a really good assistant during development.

In the past, you needed to create or do something. You would search in search engines, read documentation, and study forums.

And now? It's enough to set the right prompt and get a result with comments. In the most open technologies, the answer often turns out to be the most relevant. Although, let's be honest, not always. For example, from the questions I asked about C# WinUI3, I never received a clear working answer. Therefore, in specific areas where deep knowledge of the subject is important and the content is not widely spread, such systems are more likely to harm than help.

So don't try to operate on ideas by reading ChatGPT notes.

And here we come to the most important thing, namely the conflict of interest between companies developing LLMs and neural networks and the content creators on which these LLMs are trained.

For example, today I saw the news that Disney is going to sue Midjourney.

And I, for example, do not blame Disney, because all these AI companies are essentially freeloaders that scan hundreds of resources a day, train their neural networks, and do not pay anything to content creators, yet charge for using their services.

I personally use the paid API from OpenAI, but it bothers me a bit that OpenAI and other AI companies download my content and don't even pay for it. After all, the same Google gave and gives priority in search to sites where the original content appeared first, while ChatGPT won't even give you a link, even if the information it provided to the user is only on your resource. And all SEO optimization was about making the most original, useful, and promoted content that other resources would link to.

I know that many EVILEG users found information here that wasn't available anywhere else. This is what helped me significantly change my life because I was noticed. And now many resources may fade into the shadow of LLM.

Yes, it's progress, and those who don't adapt will simply disappear. But that doesn't mean we won't fight and develop, especially at the expense of those who train their neural networks on our content.

So, we need to grab some popcorn and beer and watch this.

In the meantime, I suggest blocking a dozen AI bad actors in nginx.

List of AI bots I've already blocked

1. GPTBot — a bot from OpenAI for data collection to train language models (ChatGPT).
2. OAI-SearchBot — another OpenAI crawler used for analyzing public web content.
3. PhindBot — a bot from the Phind search engine, focused on programmers and AI assistance.
4. SBIntuitionsBot — a bot from Scale AI, specializing in annotations and data for AI.
5. QuillBot — a bot from a paraphrasing and text generation service based on AI.
6. MistralAI — a crawler from the European language model developer Mistral AI.
7. PerplexityBot — a bot from the Perplexity.ai search engine, using AI to generate answers.
8. Perplexity-User — a user agent related to access through Perplexity.ai.
9. Google-CloudVertexBot — a bot from Google Cloud Vertex AI, a platform for ML development.
10. Google-Extended — an agent related to the use of content for AI from Google (including Bard/AI).
11. FirecrawlAgent — a bot from Firecrawl.ai — a creative search AI and web crawler.
12. Ai2Bot-Dolma — a crawler from the Allen Institute for AI (AI2), used for open datasets.
13. AI2Bot — another bot from the Allen Institute for AI.
14. cohere — a crawler from Cohere, developing LLM and NLP API.
15. Diffbot — a bot from a company that uses AI to structure data from the internet.
16. DuckAssistBot — a crawler from DuckDuckGo, related to the DuckAssist AI feature.
17. Claude — a user agent from Anthropic, related to the Claude model.
18. CCBot — a bot from Common Crawl, whose data is actively used for AI training.
19. ChatGPT-User — traffic coming from ChatGPT users, for example, when opening links.
20. Bytespider — a bot from ByteDance, used in AI research and modeling.
21. Brightbot — an AI bot with little information, but presumably related to generative AI.
22. aiHitBot — a bot aggregating business information, using AI for data processing.
23. bedrockbot — an agent from Amazon Bedrock, a generative AI platform.
24. anthropic-ai — a bot from Anthropic, developer of the Claude language model.

How to block AI bots at the Nginx level

map $http_user_agent $bot_type   {
    default 0;
    # approved bots
    "~*Some approved bot"       1;
    # bad bots
    ~*GPTBot                   2; # OpenAI — data collection for ChatGPT
    ~*OAI-SearchBot            2; # OpenAI — analysis of public web content
    ~*PhindBot                 2; # Phind — search engine with AI assistant
    ~*SBIntuitionsBot          2; # Scale AI — annotation and data for AI
    ~*QuillBot                 2; # QuillBot — AI for text paraphrasing
    ~*MistralAI                2; # Mistral AI — language models
    ~*PerplexityBot            2; # Perplexity.ai — AI search
    ~*Perplexity-User          2; # Perplexity.ai — user access
    ~*Google-CloudVertexBot    2; # Google Cloud Vertex AI — ML platform
    ~*Google-Extended          2; # Google AI — content collection for Bard, etc.
    ~*FirecrawlAgent           2; # Firecrawl.ai — AI crawler
    ~*Ai2Bot-Dolma             2; # AI2 (Allen Institute) — open data collection
    ~*AI2Bot                   2; # AI2 — AI projects and datasets
    ~*cohere                   2; # Cohere — language models and NLP
    ~*Diffbot                  2; # Diffbot — data structuring with AI
    ~*DuckAssistBot            2; # DuckDuckGo — DuckAssist AI feature
    ~*Claude                   2; # Anthropic — Claude model
    ~*CCBot                    2; # Common Crawl — data for AI training
    ~*ChatGPT-User             2; # OpenAI — traffic from ChatGPT users
    ~*Bytespider               2; # ByteDance — AI and data collection
    ~*Brightbot                2; # AI bot, presumably generative AI
    ~*aiHitBot                 2; # AI aggregator of business data
    ~*bedrockbot               2; # Amazon Bedrock — generative AI
    ~*anthropic-ai             2; # Anthropic — Claude language model
}
map $bot_type $is_human       { 0 1; default 0; }
map $bot_type $is_approved    { 1 1; default 0; }
map $bot_type $is_bad         { 2 1; default 0; }
server {
    server_name your.server.com;
    access_log /path/to/your/website/logs/access.log combined if=$is_human;
    access_log /path/to/your/website/logs/access-approved-bots.log combined if=$is_approved;
    access_log /path/to/your/website/logs/access-bots.log combined if=$is_bad;
    # 🚫 Block Bad Bots
    if ($bot_type = 2) {
        return 403;
    }
    # Another nginx logic
}