mcbal
✨ TL;DR: We design a transformer-shaped module built around relaxation steps of a driven disordered spin system after an input “quench”. Taking one learned-initialization step $\approx$ a parallel transformer block, taking $\infty$ steps $\approx$ a deep-equilibrium module. A toy language-modeling training experiment shows that the one-step modules learn to use task-relevant transient…
Introduction ✨ TL;DR: We interpret and implement transformer modules as driven, disordered vector-spin models whose response behavior can be shaped by learning parameterized interactions, gradually steering a cascade of near-equilibrium steady-state magnetizations towards solving a given objective. Using dynamical mean-field theory, we show that a first-order approximation of the update equations…
✨ Update (April 2023): Consider reading where we continue building on the intuition of probing a spin system to engineer its collective response but get rid of the assumption of symmetric coupling matrices by shifting focus from equilibrium free energies to dynamical mean-field approximations of non-equilibrium vector-spin models. Introduction In this post, we try to distill a unifying perspective…
✨ Update (November 2021): Consider reading for a high-level overview of some of the ideas outlined in this post. Introduction ✨ TL;DR: We consider transformer modules as wrappers around a differentiable steepest-descent approximation of simple Ising-like vector-spin models familiar from statistical mechanics. We observe that a blueprint of the successful transformer-like architectural pattern of…
✨ Update (November 2021): Consider reading for a high-level overview of some of the ideas outlined in this post. Introduction ✨ Code: A reference PyTorch implementation of the ideas outlined in this blog post is available in the repository . Comments welcome. To explore progress beyond the cage of softmax attention, we have previously looked at energy-based perspectives on attention mechanisms:…
Introduction 📓 Colab notebook available . Comments welcome. Recent work 1 2 has shown that the softmax-attention update step in transformer models can be intepreted as a one-step gradient update or “inference” step of a judiciously chosen energy function. An overview of these ideas can be found in previous blog posts: The goal of this educational blog post is to explicitly show how…
✨ Update (November 2021): Please consider reading for an arguably more comprehensive approach towards understanding transformers from a physics perspective. _In your [previous post](https://mcbal.github.io/post/an-energy-based-perspective-on-attention-mechanisms-in-transformers/), you introduced the energy function of modern Hopfield networks without explanation. Where does it come from? What's up…
XKCD 793: A physicist encountering machine learning for the first time ✨ Update (November 2021): Please consider reading for an arguably more comprehensive approach towards understanding transformers from a physics perspective. Introduction In 2017, 1 demonstrated state-of-the-art performance in neural machine translation by stacking only (self-)attention layers. Compared to recurrent neural…