Weight-Init Conditioned Bayesian Neural Network Priors
An alternative Bayesian neural network prior, that we might believe a little more - but that sadly doesn’t work very well.
Maxime Robeyns' Articles
Latest posts
Bayesian Low-Rank Adaptation for Large Language Models
An overview of some recent work, published in ICLR 2024, where we estimate the uncertainty and marginal likelihoods in LLMs using Bayesian LoRA adapters. We focus on the fine-tuning setting, and scale our method to LLMs using a Laplace approximation with low-rank K-FAC.
Second-Order Methods in Machine Learning
A motivation of the Hessian from an optimisation perspective (and the related Generalised Gauss-Newton / Fisher Information Matrix), an introduction to Kronecker-factored approximate curvature, and applications of the curvature in machine learning.
Of VJPs and JVPs
Some intuitions and visualisations of vector-Jacobian products and Jacobian-vector products, to help you avoid confusing the two again.
Fixing LLM Defects and Adding Skills with Synthetic Data
A note on fine-tuning transformer language models on synthetically generated training data.
Scaling Gaussian Processes
An overview of approximation methods and computational techniques for scaling Gaussian processes to large, high-dimensional datasets; covering training conditionals and variational approximations.
Bayesian Linear Regression
A review of the basic methods behind Bayesian linear regression, as well as modern techniques for approximate inference, dealing with non-conjugate priors and scaling this model to large datasets.
The Challenges of Diffusion on the Probability Simplex
An overview of some recently proposed methods for using diffusion models with discrete data, and some associated challenges.
Bayesian Flow Networks, with Code
An explanation of the recently published Bayesian Flow Networks and a PyTorch implementation.
Piecewise Linear Log-Likelihood Estimation
Or “PL3E” for short; a versatile likelihood defined by a product of piecewise-linear log-likelihood functions.