Recent content on Max Halford
The last few years Although my PhD had nothing to do with online machine learning, that’s when I started to take an interest in it. I made a Python package called creme in 2019. We joined forces with scikit-multiflow in 2020 and agreed on the name River . For some reason, I had this romantic vision of people working together in roughly equal proportions. However, I feel I did most of the…
I am in favor of coding agents. They’re here to stay, so we might as well figure out enjoyable ways to work with them. I’m aligned with Charity Majors’ Make AI Boring Again piece. I’m also increasingly convinced they won’t replace techies, because the non-techies still need us. I stumbled on this Slack thread at work: At work, we log the hours we spend on each…
Introduction I joined Carbonfact as a first employee. In a nutshell, it’s software that takes in raw ERP and PLM data, and spits out environmental reports. I got pulled into many Sales processes and customer onboardings, in addition to building the software. As the company grew, so did my expertise, which made me relevant for selling to and managing big logos. We signed a lot of well-known…
Andrej Karpathy’s autoresearch project made a lot of ripples. I guess it’s exciting because a program that modifies itself feels like actual AI. The concept of autonomous programs is not new – see genetic programming – but what we’re witnessing now is somewhat more convincing. We just got a bit closer to Neuromancer’s world, with a Turing Police that’s in…
Many people seem to admire DuckDB. But most of us are stuck with our traditional warehouses, because they’re entrenched in our data stacks and IT landscape. This is with good reason: BigQuery, Snowflake, ClickHouse and co. are great software. But they’re not cheap, and keeping a warehouse’s monthly bill under control is non-trivial. What if you could get the best of both worlds?…
Python 3.14 introduced the compression.zstd module. It is a standard library implementation of Facebook’s Zstandard (Zstd) compression algorithm. It was developed a decade ago by Yann Collet, who holds a blog devoted to compression algorithms. I am not a compression expert, but Zstd caught my eye because it supports incremental compression. You can feed it data to compress in chunks, and it…
Détrak is a simple board game. There’s a 5x5 grid, and players need to place 12 domino-like pieces to cover the grid completely. The domino symbols are determined by rolling two dice. The dice are six-sided, and their rolls are shared by all players. Points are scored based on adjacencies of matching symbols. The goal is to find the optimal placement of the pieces to maximize the score based…
At my day job we deal with enterprise customers. They pay us a subscription fee, and in return we help them in various ways to reduce their carbon footprint. To keep the boat afloat, we need to make some money. We shouldn’t spend more money than we make. So we need to keep track of our revenue and costs. Our gross margin is (revenue - cost) / revenue , where the cost is mostly the salaries…
I have a friend with whom I like to go travelling on a bicycle. We usually go for a couple of weeks, and we’ve been doing it regularly over the past five years. We’ve cycled in France and England so far. Both countries have many affordable campsites, and there’s always a decent hotel/Airbnb not too far for when it’s raining. But this year we travelled through Switzerland,…
Spoiler: not really dafont.com is a wonderful website that contains a large collection of fonts. It’s more comprehensive and esoteric than Google Fonts. One of its features is a forum where users can ask for help identifying fonts – check out this poor fellow who’s been waiting for over two years and bumped his thread. I thought it would be interesting to see if an LLM could do…
DuckLake is the new data lake/warehouse from the makers of DuckDB. I really like the direction they’re taking. I’m hopeful it has the potential to streamline the data engineering workflow for many people, vastly reducing costs along the way. I’m a bit of a nut and don’t use SQLMesh or dbt. Instead, I built lea a few years ago, and we still use it at Carbonfact . I would…
A metric tree is a visual way to organize a complex metric. Count gives a good introduction here . Abhi Sivasailam gave a popular talk at Data Council 2023 if watching videos is your thing. Ergest Xheblati is someone to follow if you want to go deeper. There’s also a recent article from Lightdash. Finally, there’s this article by Timo Dechau, but it’s behind a paywall. The…
I recently looked into reducing the runtime of Carbonfact ’s SQL DAG. Our DAG is made up of roughly 160 SQL queries. It takes about 10 minutes to run with BigQuery, using on-demand pricing. It’s decent. However, the results of our DAG feed customer dashboards, and we have the (bad) habit of refreshing the DAG several times a day. Reducing the runtime by a few minutes can be a nice…
Carbonfact’s customers are clothing brands and factories. Our mission is to measure (and ultimately reduce) the carbon footprint of their products. We need primary data to do this: purchase orders, bills of materials, energy consumption data, etc. Each customer has a unique IT setup, which makes it challenging to scale to hundreds/thousands of customers. Our success as a business depends on…
I spend a lot of time at Carbonfact working on datasets shared by our customers. We typically set things up so that our customers can export data automatically. They usually deposit files to a GCP bucket, with a script, once a day. We then have an ETL script for each customer that runs afterwards to fetch their latest data and process it. During development, I load customer data to my laptop and…
Measuring the environmental impact of a product is done using life cycle assessment (LCA). This is a methodology that breaks down a product’s life cycle into stages ( LCI ), and measures the impact of each stage on the environment ( LCIA ). There are a few pieces of LCA software to choose from. The leading ones are SimaPro , GaBi , openLCA , and Umberto . These are all proprietary software,…
Our mission at Carbonfact is to measure the environmental impact of clothes. This involves a lot of steps. The main one is to determine what materials a product is made of, along with each material’s mass. This is straightforward for most clothes like jumpers and pants. These are typically made of a single fabric, such as cotton or polyester. The mass of each material is roughly the same as…
Last night I went to a Toulouse Data Science meetup. The talks were about generative AI and information retrieval, which aren’t topics I’m knowledgeable about. However, one of the speakers was a friend of mine, so I went to support him. Toulouse used to be my hometown, so I bumped into a few people I knew. It was a nice evening.
This is a short post to share a GitHub Actions pattern I use to setup Poetry and pre-commit . These two tools cover most of my Python development needs. I use Poetry to manage dependencies and pre-commit to run code checks and formatting. The setup is fast because it caches the virtual environment and the .local directory. I like to use custom actions for this type of stuff. These are base actions…
Funnel metrics as products I talked about metric decomposition in a previous article , and how it can be used to explain why metrics change values over time. That article explained how to decompose a sum, as well as a ratio. In this article, I’ll explain how to decompose a product. revenue = impressions * click_rate * conversion_rate * spend The decomposition in this article isn’t…
A tenant of the modern data stack is the use of ELT (Extract, Load, Transform) over ETL (Extract, Transform, Load). In a nutshell, this means that most of the data transformation is done in the data warehouse. This has become the de facto standard for modern data teams, and is epitomized by dbt and its ecosystem. It’s a great time to be a data engineer! We at Carbonfact fully embrace the ELT…
I’m writing this after watching the talk Joe Reis gave at Big Data LDN . It’s called Data Modeling is Dead! Long Live Data Modeling! It’s an easy-to-watch short talk that calls out on a few modern issues in the data world. I’d like to bounce off one of Joe’s slides: I’m aligned with Joe that many issues stem from the lack of unison between engineering and data…
Edit – I published a notebook here that deals with the case where dimension values may (dis)appear from one period of time to the next. The notebook decomposes a ratio, but the logic is also valid for decomposing a sum. Edit 2 – I’ve stumbled on this article by Shao Zhifei which provides a good derivation of the ratio decomposition formula. I contacted Shao Zhifei on LinkedIn,…
Making environmentally friendly decisions can only be done with the right information. At Carbonfact, we’ve realized a big challenge is the lack of information about industrial processes. We tackle that slowly but surely by gathering data from various sources, and making it available to our customers. Regarding food, the French government has a great initiative called Agribalyse . It’s…
Introduction Graph problems are quite common. However, it’s rare to have access to a database offering graph semantics. There are graph databases, such as Neo4j and GraphX , but it’s difficult to justify setting one of those up. One could simply use networkx in Python. But that only works if the graph fits in memory. From a practical angle, the fact is that people are querying data…
I love the JSON format. It’s the kind of love that grows on you with time. Like others, I’ve been using JSON everywhere for so many years, to the point where I just take it for granted. I suppose the main thing I like about JSON is its flexibility. You can structure your JSONs without too much care. There will always be a way to consume and manipulate it. But I have discovered a bit of…
According to the United Nations, the 15th of November was the day we crossed 8 billion humans on the planet. How can they be so sure of that? Surely there has to be some margin of error, meaning it could have happened on the 14th or 16th. Then again, does it matter? I would argue almost all metrics we look at are incorrect. For instance, I work at a company whose goal is to measure the carbon…
Edit – this post generated a few insightful comments on Hacker News. I’ve also put the code in a notebook for ease of use. Introduction Modern MLOps is complex because it involves too many components. You need a message bus, a stream processing engine, an API, a model store, a feature store, a monitoring service, etc. Sadly, containerisation software and the unbundling trend have…
A program which compiles and runs without errors isn’t necessarily correct. I find this to be especially true for statistical software, both as a developer and as a user. Small but nasty bugs creep up on me every week. I keep sane in the membrane by writing many unit tests 🐛🔨 I make heavy use of doctests . These are unit tests which you write as Python docstrings . They’re really…
Active learning is a way to get humans to label data efficiently. A good active learning strategy minimizes the number of necessary labels, while maximizing a model’s performance. This usually works by focusing on samples where the model is unsure of its prediction. In a batch setting, the model is periodically retrained to learn from the freshly labeled samples. However, the training time…
TLDR I compared the energy consumption of Airbnb guests versus their host, in the same apartment, during 2022. It appears that guests do in fact consume more energy than hosts. The data I used is available to any Airbnb host. I also open-sourced all the code I wrote for this analysis. Introduction European energy prices have soared in 2022. It’s gone to the point where some Airbnb hosts have…
Source When I see tweets like this one, I’m both happy because people are aware of River , but also irked because it’s really difficult to make production-grade open source software. We just had a developer meeting a week ago. We planned what we will work on during the first half of 2023. I thought it would be worthwhile to give a high-level view of how we envision River’s…
The task You’ll have heard of GPT-3 if you haven’t been hiding under a rock. I’ve recently been impressed by Nat Friedman teaching GPT-3 to use a browser, and SeekWell generating SQL queries from free-text. I think the most exciting usecases are yet to come. But GPT-3 has a good chance of changing the way we approach mundane tasks at work. I wrote an article a couple of months…
This article has some interactive keyboards, therefore I recommend reading it from your computer rather than your phone. On-screen TV keyboards I’ve recently been spending time at my brother’s place. We usually eat in front of TV. I’ve thus found myself typing stuff on the Netflix/Amazon/Plex TV apps. The typing happens through a remote controller, which is slower than typing…
The task I work at a company called Carbonfact . Our core value proposal is computing the carbon footprint of clothing items, expressed in carbon dioxide equivalent – $kgCO_2e$ in short. For instance, we started by measuring the footprint of shoes – no pun intended. We do these measurements with life cycle analysis (LCA) software we built ourselves. We use these analyses to fuel…
The inverse covariance matrix, also called precision matrix , is useful in many places across the field of statistics. For instance, in machine learning, it is used for Bayesian regression and mixture modelling . What’s interesting is that any batch model which uses a precision matrix can be turned into an online model. That is, provided the precision matrix can be estimated in a streaming…
Disclaimer Let me be absolutely clear: I think dbt is a great tool. Although this post is a rant, the goal is to be constructive and suggest an improvement. dbt in a nutshell dbt is a workflow orchestrator for SQL. In other words, it’s a fancy Make for data analytics. What makes dbt special is that it is the first workflow orchestrator that is dedicated to the SQL language. It said out loud…
River is a Python software for doing online machine learning. It’s the result of a merger in early 2020 between creme and scikit-multiflow . Saulo Mastelini , Jacob Montiel , and myself are the three core developers. But there are many more people who contribute here and there! This week Saulo Mastelini and I got to meet in person. This is worth mentioning because Saulo is originally from…
Fuzzy string matching in a nutshell Say we’re looking for a pattern in a blob of text. If you know the text has no typos, then determining whether it contains a pattern is trivial. In Python you can use the in function. You can also write a regex pattern with the re module from the standard library. But what about if the text contains typos? For instance, this might be the case with user…
I recently saw SymSpell pop up on Hackernews. It claims to be a million times faster than Peter Norvig’s spelling corrector . I think it’s great that there’s a fast open source solution for spelling correction. But in my experience, the most challenging aspect of spelling correction is not necessarily speed. When I worked at Alan , I mostly wrote logic to extract structured…
Edit (2023-05-26) – I’ve learnt about the Kumiko project , which is exactly devoted to slicing comic book panels. There’s even a live tool . I discovered it thanks to being pinged on this issue. Motivation I’ve recently been reading some comic books I used to devour as a kid. Especially those from the golden era of francophone comics: Thorgal, Lanfeust, XIII, Tintin, Largo…
Why I’m writing this Fact: designing open source software is hard. It’s difficult to make design decisions which don’t make any compromises. I like to fall back on Dieter Rams’ 10 principles for good design . I feel like they apply rather well to software design. Especially when said software is open source, due to the many users and the plethora of use cases. I had to make…
I’m working on a problem where I need to sample k items from a list without replacement. The sampling has to be weighted. In Python, numpy has random.choice method which allows doing this: import numpy as np n = 10 k = 3 np . random . seed ( 42 ) population = np . arange ( n ) weights = np . random . dirichlet ( np . ones_like ( population )) np . random . choice ( population , size = k ,…