Raphael Pisoni's personal blog.
I swapped the transformer's matchmaking service for a handful of Gaussians, and it beat content scoring without comparing a single pair of tokens. Crack open any vision transformer and you'll find the same machinery humming at its core: scaled dot-product attention (SDPA). Every token asks every other token "how much do I care about you?", and the answer is a big $N \times N$ matrix of similarity…
The standard Cross-Entropy loss has a well-known flaw: it forces neural networks to push their features toward infinity just to reach full confidence. The result is a messy latent space and models that confidently hallucinate when fed absolute garbage data. This post breaks down my attempt to fix this with the HALO-Loss. It covers the weird geometry of high-dimensional "soap bubbles", how wiring a…
If you crack open the architecture of almost any modern Transformer, you will find Scaled Dot-Product Attention (SDPA) sitting at its core. We rarely second-guess it. It is heavily optimized by hardware accelerators, it scales beautifully, and empirically, it runs the world. But if you look closely at the underlying math, treating a dot product as a proxy for "similarity" carries some subtle…
The "Infinite Gap" and Why Softmax Keeps Me Up at Night To understand any solution, we first have to really understand the problem. I've spent the better part of my research career staring at loss curves, watching them dip, plateau, and occasionally spike catastrophically. We often treat the loss function as a black box, a simple signal telling the network "good dog" or "bad dog." But if you look…
Classification with neural networks is weird! There, I said it! We usually have a single output per class, as if for some reason each class was it's own feature. The numbers these outputs produce are then intepreted as a log-probability distribution over all the available classes. Eveybody knows it doesn't make sense, yet we treat it as a mathematical assumption. Also needing a separate output for…
A lot has happened since my last post on the Sharpened Cosine Similarity layer. In this post I will try to give you a quick overview over the most important developments around this feature extractor that shapes up to be more and more interesting. Read more… (4 min remaining to read)
Some days ago Brandon Rohrer retweeted his own twitter thread from 2020 in which he makes the argument that convolutions are actually pretty bad at extracting features. In it he proposes a method to improve feature extraction that seemed compelling to me. The formula for this Sharpened Cosine Distance is the following: $$ scd(s, k) = sign(s \cdot k)\Biggl(\frac{s \cdot…
CLIP is a model published by OpenAI that is able to learn visual concepts by natural language supervision. It does this by embedding images and their corresponding caption into a joint space and contrastively minimizing their distance. OpenAI only published weights for CLIP trained on english data. That's why during the JAX/Flax community event organized by Hugging Face and Google, we from the…
Image augmentations make all the difference when working with neural networks. Everybody should know that by now. No matter what you're trying to train, if it involves images you should be using heavy and fancy augmentations! The only downside of these heavy augmentations is that they might slow down your training significantly if they are not implemented in a fast and efficient way. With Imax the…
Since my paper on "Joint Unsupervised Depth-Estimation and Obstacle-Detection" did not get accepted to NeuRIPS 2019 I now had another unpublished paper lying around. Back then more and more people around me started to get interested in neural networks and some (including my mom😂) were also interested in my work. I however, kept struggling trying to explain to them what exactly it was I was…