The Problem
On June 5th, my personal instance of Gotosocial felt really slow. I tried restarting the docker container, but that didn’t help. I checked what was happening in htop and CPU utilization was 100% for all 4 cores. Bots were scraping 7MB/s in data. My blog was getting hammered for some reason.
Through lots of trial and error I tried many things to fight it. Here’s what helped me.
The Solutions
- iocaine to feed AI garbage data to scrapers (already was doing this)
- use caddy-defender mod to block known scrapers
- use rate_limit mod
- robots.txt to block scrapers
My Caddyfile
(caddy-common) {
encode gzip
header {
-Server
Strict-Transport-Security "max-age=31536000; include-subdomains;"
X-XSS-Protection "1; mode=block"
X-Frame-Options "DENY"
X-Content-Type-Options nosniff
Referrer-Policy no-referrer-when-downgrade
X-Robots-Tag "none"
}
}
# Iocaine - AI tarpit, reusable code snippet
(iocaine) {
@read method GET HEAD
reverse_proxy @read 127.0.0.1:42069 {
@fallback status 421
handle_response @fallback
}
}
# block AI user-agents
(block_bots) {
# https://github.com/ai-robots-txt/ai.robots.txt
@aibots {
header_regexp User-Agent "(AddSearchBot|AgentTimes|AI2Bot|AI2Bot\-DeepResearchEval|Ai2Bot\-Dolma|aiHitBot|amazon\-kendra|Amazonbot|AmazonBuyForMe|Amzn\-SearchBot|Amzn\-User|Andibot|Anomura|anthropic\-ai|ApifyBot|ApifyWebsiteContentCrawler|Applebot|Applebot\-Extended|Aranet\-SearchBot|atlassian\-bot|Awario|AzureAI\-SearchBot|bedrockbot|bigsur\.ai|Bravebot|Brightbot|Brightbot\ 1\.0|BuddyBot|Bytespider|CCBot|Channel3Bot|ChatGLM\-Spider|ChatGPT\ Agent|ChatGPT\-User|Claude\-Code|Claude\-SearchBot|Claude\-User|Claude\-Web|ClaudeBot|Cloudflare\-AutoRAG|CloudVertexBot|Code|cohere\-ai|cohere\-training\-data\-crawler|Cotoyogi|Crawl4AI|Crawlspace|Datenbank\ Crawler|DeepSeekBot|Devin|Diffbot|DuckAssistBot|Echobot\ Bot|EchoboxBot|ExaBot|FacebookBot|facebookexternalhit|Factset_spyderbot|FirecrawlAgent|FriendlyCrawler|Gemini\-Deep\-Research|Google\-Agent|Google\-CloudVertexBot|Google\-Extended|Google\-Firebase|Google\-Gemini\-CLI|Google\-NotebookLM|GoogleAgent\-Mariner|GoogleOther|GoogleOther\-Image|GoogleOther\-Video|GPTBot|HenkBot|iAskBot|iaskspider|iaskspider/2\.0|IbouBot|ICC\-Crawler|ImagesiftBot|imageSpider|img2dataset|ISSCyberRiskCrawler|kagi\-fetcher|Kangaroo\ Bot|KlaviyoAIBot|KunatoCrawler|laion\-huggingface\-processor|LAIONDownloader|LCC|LinerBot|Linguee\ Bot|LinkupBot|Manus\-User|meta\-externalagent|Meta\-ExternalAgent|meta\-externalfetcher|Meta\-ExternalFetcher|meta\-webindexer|MistralAI\-User|MistralAI\-User/1\.0|MyCentralAIScraperBot|NagetBot|netEstate\ Imprint\ Crawler|newsai|NotebookLM|NovaAct|OAI\-SearchBot|omgili|omgilibot|OpenAI|opencode|Operator|PanguBot|Panscient|panscient\.com|Perplexity\-User|PerplexityBot|PetalBot|PhindBot|Poggio\-Citations|Poseidon\ Research\ Crawler|QualifiedBot|QuillBot|quillbot\.com|SBIntuitionsBot|Scrapy|SemrushBot\-OCOB|SemrushBot\-SWA|Shap\-User|ShapBot|Sidetrade\ indexer\ bot|Spider|TavilyBot|Terra\ Cotta|TerraCotta|Thinkbot|TikTokSpider|Timpibot|Trae|TwinAgent|VelenPublicWebCrawler|WARDBot|Webzio\-Extended|webzio\-extended|wpbot|WRTNBot|YaK|YandexAdditional|YandexAdditionalBot|YouBot|ZanistaBot)"
}
}
# global caddy options
{
# to log metrics for prometheus
metrics {
per_host
}
order rate_limit after basic_auth
order defender before basic_auth
}
# my blog which was under attack
blog.arkadi.one {
# block bots with robots.txt
# import brings in the code at the top with (NAME)
import block_bots
# block AI traffic
defender block {
ranges aliyun deepseek githubcopilot openai mistral aws vultr gcloud cloudflare digitalocean linode
}
# slow down all other traffic
defender ratelimit {
ranges all
}
rate_limit {
match header X-Defender-RateLimit true
rate 1r/s
}
import iocaine
root * /var/www/html
file_server
log {
output file /var/log/caddy/blog.arkadi.one.log
}
}
NOTE: To use caddy-defender and rate_limit you need to build custom Caddy executable with xcaddy.
This is the script I use to do that.
echo "OMG. CADDY GOT UPDATED!"
xcaddy build --with github.com/caddy-dns/gandi --with github.com/mholt/caddy-ratelimit --with pkg.jsn.cam/caddy-defender
sudo chown root:root caddy
sudo mv caddy /bin/
echo "CADDY IS NOW UPDATED!"
Defence In Depth
After incorporating all those defenses on my site, my CPU usage for caddy is down to 3% which is normal. Also, I’m only serving 100-200KB/s of data.
Not one thing will probably help you. You probably need to layer your defences using multiple things. Good luck! Share tips and tricks with others on the fediverse.
I’m still not sure why my blog was targeted. I’m not done with this. I have to keep an eye on this and continue to harden my websites. I am currently thinking about how I can import caddy metrics into prometheus to bring it into Grafana.
The battle continues.