Block the Bots

The Crawl Ratio Problem The agentic internet is on the rise. In 2025 Q2, the visitation ratio for human to AI bot traffic was 50:1. By Q4, AI bot traffic increased with a ratio of 31:1 ( 1 ) . With 4.2% of all HTTP internet traffic originating from AI bots in 2025 ( 2 ) , and crawl-to-refer ratios looking abysmal ( 3 ) (Anthropic @ 41.1K:1 & OpenAI @ 1.1K:1), the internet is increasingly becoming…