Improving legacy code: Using Task Queue to Speed Up a Crawler in an ETL Process

In this post, I will present how I improved an ETL process to be more than 4x faster using the same resources. While the old architecture used Python threads, the new one used task queue architecture to be more reliable and scalable. So, I will explain how we can improve a legacy code and speed it up by only modifying the architecture to run the code. &#xA; Almost 10 years ago, I was hired to…