Article Summary: Hidden Technical Debt in Machine Learning Systems

Link to the original article, written in 2015 by 10 Googlers.

My main takeaways -

1. The code that builds the ML model makes about 5% of the total code of an ML system in production. This is an example of all the code that is required in such an infrastructure:

2. Patching up an infrastructure to quickly get to production is never a good practice, and ML infra is not different.

3. Some of the issues to be aware of when planning an infrastucture, that are unique to ML infra are: coupling of input/output between the different models in the system, direct or indirect. At this point there is no tooling to catch this.

4. More on that point - dead experiments that are zombieing around can still be acting as inputs, although no one is aware. So pruning or officially integrating experiments is extra important.

5. Configurations are particularly close to my heart. An ML infra has all the regular configurations, and in addition configurations that are special to the model(s).

6. Python is the default language for ML (In my opinion it doesn’t have to be the default, but that’s for another post). If the rest of the code is written in another language - treating the infrastructure as microservices becomes something you have to do.

7. Data and biases - can’t talk about ML without talking about this point. “The distribution of predicted labels is equal to the distribution of observed labels“ is a monitoring practice recommended in the article. The benefit is that once reality changes - you can easily notice with this monitoring set up.