Blocking the Scrapy bot

Here's the reason I decided to take a closer look at who is trying to scrape my site. It's Scrapy, a popular Python framework often used for web scraping and data collection. I might not have paid attention to this, but when your site is shamelessly loaded to 100%, you start thinking about banning the culprit. That's exactly what I did. In nginx, it might look like this if ($http_user_agent ~*…