RSSAmplifier

Blog

Alden Page's Tech Blog

alden.pageRSS feed ↗2 posts

Latest posts

Turn Your Dead Personal Blog Into a Tor WebTunnel in 5 Minutes

Like many people in the software world, I have a personal website that I have zero time to maintain. Outside of hosting a vanity email address and an old whitepaper , there's not a whole lot going on here. My VPS spends 99% of its cycles idling. The other 1% is probably being spent on serving drive-by WordPress vulnerability probes and crawl bots trawling for training data . With just a few…

How to Politely Crawl and Analyze Half a Billion Images

Introduction A few years ago, Creative Commons tasked me with building a web crawler capable of downloading 500 million images. Crawling anything beyond a few thousand URLs demands a fast distributed system. Moreover, it's not enough to be fast; moral, legal, and practical considerations demand that a crawler be polite : a crawler must be carefully designed to avoid exhausting the resources of its…