This site does not allow itself to be embedded. You can still read it on the original site — the toolbar below keeps your place in the directory.
Here's the reason I decided to take a closer look at who is trying to scrape my site. It's Scrapy, a popular Python framework often used for web scraping and data collection. I might not have paid attention to this, but when your site is shamelessly loaded to 100%, you start thinking about banning the culprit. That's exactly what I did. In nginx, it might look like this if ($http_user_agent ~*…
Here's the reason I decided to take a closer look at who is trying to scrape my site. It's Scrapy, a popular Python framework often used for web scraping and data collection. I might not have paid attention to this, but when your site is shamelessly loaded to 100%, you start thinking about banning the culprit.
That's exactly what I did.
In nginx, it might look like this
if ($http_user_agent ~* "Scrapy") {
return 403;
}
The article first appeared on EVILEG - Practical programming
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.