Thoughts on Math and ML
From One Agent to Many The math behind multi-agent ensembles is simpler than you think – and it tells you exactly when adding agents helps. You’ve built an AI agent that investigates anomalies. A detection rule fires – unusual login location, impossible travel, a service account calling an API it’s...
No generative AI here - just good old-fashioned statistics. This post introduces Bayesian networks: visual models that connect the clues you see to the threats you care about. Instead of drowning in isolated alerts, a Bayesian network lets your SOC update its belief about an attack as new evidence arrives....
Every system has properties that define its usefulness. We talk about efficiency when we care about speed and resource usage. We talk about stability when we care about consistency under load. We talk about usability when we care about how easy it is for a human to interact with the...
Ever had a complex Python function you wanted to quickly host as an API? Managing dependencies like databases, secret keys, or heavy utilities like audio processing libraries can be quite cumbersome. Typically, I handle this with a pattern involving: Docker: to manage and install dependencies AWS Parameter Store: to securely...
When building systems to detect cyberattacks or fraud, technical expertise alone isn’t enough. Success hinges on adopting a detection mindset—a way of thinking that prioritizes skepticism, rapid iteration, and rigorous validation. A detection mindset ensures that detection engineers and data scientists remain vigilant, adaptive, and results-driven in their approach. Here’s...
Software engineering is changing. From autocomplete to chat-based code generators to no-code platforms, AI tools arrived like a tsunami in 2024. It’s true that most software development work - especially for experienced engineers working on distributed systems and mature codebases - remains above the waterline. Smart autocomplete and agentic code...
Asynchronous communication through tools like Email, Slack, and Docs has become central to how teams work. But how do we ensure clarity when we can’t rely on real-time feedback? Words and sentences can vary widely in meaning, and the bedrock of effective communication is shared context. In a conversation context...
Suppose you are a payment processor handling thousands or millions of payments every day. Some small number of the payments that you process are bound to be fraudulent. How can you spot them? This problem - identify a small number of bad events in a sea of good events -...
Sloppy analytics can lend false confidence to bad decisions. Leaders who understand a few common pitfalls can make better decisions from data. The most common pitfall is underspecification. The answer to a simple-seeming question like “are users more likely to churn if they see more than 3 ads on their...
The CEO needs to increase company margins. He turns to the engineering leadership team to help. Engineering leadership puts together a sound plan. First the objective to “Increase Company Margins” is decomposed into a collection of eng-level goals like “Reduce Tier 2 Support Hours”. Each of these eng-level goals decompose...
I’ve spent a lot of time leading large and small engineering teams over the last decade. Through this experince I’ve developed some opinions about what good leadership looks like. My central leadership tenets are: Set a Strategy Do Row-Level Inspection Set Clear Expectations Give Frequent Feedback Let Builders Build Set...
Generative large language models (LLMs) like ChatGPT are like wild beasts. They are powerful but difficult to tame. One formidable challenge lies in the cost and latency associated with these models. Compared to traditional software solutions, LLMs can operate at a slower pace, incurring higher costs along the way. Instead...
Generative large language models (LLMs) like ChatGPT will revolutionize the way we approach complex problems. Enormous amounts of custom labeled data are no longer required to train many specialized AI systems. LLMs make generalized problem-solving capabilities vastly more accessible. This has profound implications for software development, data science, and machine...
ML models are collaboration bottlenecks. Suppose your team owns a binary classification model, and you have decided that increasing this model’s recall without reducing precision is very important for the business. You’ve put 5-10 engineers on the job. The Problem What should these engineers do? There are many ways to...
One of the hallmarks of a great engineer is the ability to execute. An engineer who can execute gets things done. They crush tickets, pump out designs, spin up features, answer critical data questions, improve models, and prototype new ideas quicker than their peers. How to Execute Engineering projects are...
A common problem in machine learning is “use this function defined over this small set to generate predictions over that larger set.” Extrapolation, interpolation, statistical inference and forecasting all reduce to this problem. The Kan extension is a powerful tool in category theory that generalizes this notion. In a recent...
A boxer who only punches a bag will fail in the ring, and an ML model that only learns with clean data will fail in production. We need to let our model get punched in the face in training if we want it to perform well when distributions drift. The...
Many real world problems can be framed as regression: use a collection of features \(X\) to predict a real-valued quantity \(y\). However, this framing can obfuscate a very important detail: which kinds of mistakes are most important to avoid? Many factors can influence this, including how the predictions will be...
Introduction Many prediction problems can be framed as “given the knowledge that this sample belongs to categories \(A,B,C,\cdots,D\), predict something about this sample.” As a concrete example, suppose we would like to use linear regression to predict the value of a transaction based on a small set of categorical features...
This post is part three in a three part series on the challenges of improving a production machine learning system. Find part one here and part two here. A/B Testing When engineers and data scientists optimize machine learning systems they often focus on improving offline metrics like cross entropy, ROC-AUC,...