I revived my old mac to a self-hosted cluster
Motivation: Beyond the Dust
My personal tech blog
Motivation: Beyond the Dust
When working on machine learning pipelines, most bugs aren’t about bad models — they’re about broken plumbing.
This Christmas I wanted to set my-self a fun challenge. I wanted to create an end-to-end AI product, with model serving, with performance monitoring, with a data catalog, with propper data engineering, with orchestration, with an iOS front-end, and running only on local compute. Not to much to ask, right?
AI is evolving rapidly with new models every day. However, it is easy to get lost in the hype of new models and forget how AI will be integrated at a system-wide level.
Knowing the performance of an AI model is key to confidently delivering superior AI products. As such, it’s common practice for supervised learning models to set aside a specific dataset for evaluation during the training phase of our model.
Many developers often lean towards tools that “feel good” and argue they have good Developer Experience (DX). However, this subjective measure of “feel good” is rooted in past experiences and familiarity with similar tools, making the argument of “good DX” worthless. Which is also something ThePrimeagen have argued for. Furthermore, relying solely on what feels comfortable can hinder innovation…
Navigating the landscape of MLOps can be an overwhelming task. There are thousands of tools to explore, and understanding which problem each tool tackles can take time to grasp.