in the Dev World
Live Last read · last published · next check
Sara Hooker’s essay “On the Slow Death of Scaling” (2026) is a thought-provoking piece that deserves a careful read.1 What follows is my interpretation (or perhaps more accurately, a riff) on her arguments, where I’ll both steelman her position and push back where I think the evidence points elsewhere. Scaling...
What is the difference between active learning (and active sampling) and data filtering? And why do we treat data selection differently during training versus before training? This post explores the fundamental distinction between active dataset selection and data filtering, which we will phrase as “selection vs. rejection” using an appeal to...
This note explores a seemingly simple yet surprisingly profound example of how rational agents can diverge in their beliefs even when exposed to identical evidence. While the mathematical model we’ll examine is highly simplified, its core mechanism offers a potential lens through which to understand the complex dynamics of real-world...
Why is the Bayesian model average (BMA) often hailed as the optimal choice for rational actors making predictions under uncertainty? Is this claim justified, and if so, what’s the underlying logic? No seriously, why? 😂 Please point me to the right references on this. Below, I enumerate some naive thoughts...
In the first part of this two-part series on Function-Space Variational Inference (FSVI), we looked at the Data Processing Inequality (DPI). In this second part, we finally look at the relationship between FSVI, a method focusing on the Bayesian predictive posterior rather than the parameter space, and the DPI. We...
In information theory, the data processing inequality (DPI) is a powerful concept. Informally, it tells us that processing data cannot increase the amount of contained information. In this two-part blog post, we will explore the DPI and its applications to function-space variational inference (FSVI). The data processing inequality examines how...
In this blog post, following the previous blog post1 on “Bayesian Appropriation: General Likelihood for Loss Functions”, we will examine and better understand parts of the paper “PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks”2 (“PACTran”), which was presented as an oral at the ECCV...
In this blog post, we explore how some losses could be rewritten as a Bayesian objective using ideas from variational inference—hence, the tongue-in-cheek “Bayesian Appropriation.” This can make it easier to see connections between loss functions and Bayesian methods (e.g. by spotting similar patterns in the wild). We will first provide...
The Rao-Blackwell theorem is a fundamental theorem in statistics that offers a powerful method for improving estimators by conditioning on sufficient statistics. It is named after two statisticians, C.R. Rao and David Blackwell, who independently discovered it. The theorem is relevant in many areas of statistics, including machine learning algorithms...
Software engineering is on the brink of a revolution with the emergence of large language models (LLMs). LLMs are AI systems that have been trained on large amounts of data, allowing them to generate natural language text and source code. LLMs allow developers to specify intent using prompts, rather than...
While training of deep ensembles or BNNs, we should be able to maximize the BALD score (model disagreement metric) as a regularizer using unlabeled data to improve model diversity and active learning efficiency (or OOD detection) where it matters: for pool or evaluation set data. Given the limited capacity of...
This post introduces a family of much less expensive approximations for BatchBALD that might work well where BatchBALD works. You might have noticed that BatchBALD can be very, very slow. We can approximate BatchBALD using pairwise mutual information terms, leading to a new approximation, we call 2-BALD, or generally, following...
The paper, accepted as Long Oral at ICML 2022, discusses the (log) marginal likelihood (LML) in detail: its advantages, use-cases, and potential pitfalls, with an extensive review of related work. It further suggests using the “conditional (log) marginal likelihood (CLML)” instead of the LML and shows that it captures the...
The definition of the total variation distance can be confusing (at least to me) as it is formulated as a supremum. There is a simpler formulation. We connect the two here and provide some intuitions. The reason for this post is that recently, I was looking at some more theoretical...
We express common performance metrics, such as recall, precision and so on, for classification tasks using probabilities and examine the F1 score and simplify it to a ratio that is simpler to understand. The F1 score \(F\) is usually defined as harmonic mean of precision and recall: \[ F_1 =...
This post discusses potential failure cases of outlier exposure—when using “fake” label distributions for outliers—and presents an intellectually pleasing version of outlier exposure in latent space, treating outliers as purely negative samples from a contrastive point-of-view. About this post: I repeat my motivation from the last post: during my day-to-day,...
During my day-to-day, I read papers and procrastinate from writing my thesis, so I often come up with high-level questions that I cannot research because I don’t have the experience, time, and computing resources. The following is such a research question which—if it has not been answered by someone else...
This are my notes and observations from reading the Linear Algebra chapter of the Deep Learning book. The following notes are presented in order of value to the reader. I start with a discussion of the Moore-Penrose pseudoinverse, followed by a short reflection on “broadcasting”. Moore-Penrose pseudoinverse The Moore-Penrose pseudoinverse...
I have published a couple of posts on Medium to see how it works: A Dart REPL PoC - Hacking with Dart A well-received post about https://github.com/BlackHC/dart_repl which is a PoC interactive shell for Dart. I hope I’ve been able to restore some Karma points with the Dart team over...
Today, I want to release another bit of code from my master’s thesis. However, this time it won’t be C++ code, instead I’m going to release some LaTeX code which I used to display source code fragments with. Specifically the results look like this: You can download the accompanying example...