We’re organizing a NIPS workshop on approximate inference. It is together with Tamara Broderick, Stephan Mandt, and James McInerney—and alongside an incredible cast of seminal researchers: David Blei, Andrew Gelman, Mike Jordan, and Kevin Murphy. [ Workshop homepage ] This year, we set a theme based on what we believe are some of the most important challenges. In particular, there’s an emphasis on…
This article is written with much help by David Blei. It is extracted from a discussion paper on “Fast Approximate Inference for Arbitrarily Large Semiparametric Regression Models via Message Passing”. [link] We commend Wand (2016) for an excellent description of message passing ( mp ) and for developing it to infer large semiparametric regression models. We agree with the author in fully…
In the past few days I spent time migrating the blog from Ghost to Jekyll . The theme builds off Type Theme , and is heavily inspired by Otoro , the New York Times , and the Rosenrot . The annals-like frontpage takes cue from Paul Graham’s essays and Cosma Shalizi’s notebooks . Top: Old frontpage. Bottom: New frontpage. Hooray for math! (using KaTeX) Hooray for code snippets! (taken from Edward )…
I wrote a tutorial on inference networks on the Edward website. I normally don’t like writing outsourced blog posts, but I figure it’s better to link there than write duplicated content which may later become outdated. Thanks go to Kevin Murphy for motivating the tutorial as it is based on our discussions, and also related discussion with Jaan Altosaar.
I’ve been helping write tutorials that teach concepts such as black box variational inference in Edward. And as I’ve been editing, I’ve noticed the majority of my suggestions are about the writing rather than the code. Communication is important—arguably more important in papers than the idea itself. Communication evokes different lines of thinking about how to approach problems, and it always has…
There is a tutorial on variational auto-encoders which popped up on my arXiv radar this week. Thanks to Carl Doersch for writing this tutorial! As a researcher in this area, I believe there is sorely a need for an exposition of recent developments in variational inference. Variational inference’s application for enabling deep generative models has exploded in the past few years. This tutorial is a…
As you may (or may not) know, I’ve been busy lately spear-heading Edward , an open-source library for probabilistic modeling. It’s meant to help bridge the gap between what I view as two dichotomous approaches of statistical learning: one approach develops complex models in order to achieve the best results on a specific task; and the other approach adheres to simple models in order to understand…
One feature unique to our field is the sheer amount of online resources. It’s one of the reasons I personally got into machine learning over other disciplines, as self-learning is much more accessible. This is especially true for video resources. Part of this is because as a field we are very well integrated with computer science, and thus we’re technologically more motivated. (However, arguably…
Yingzhen Li, Jose Miguel Hernandez-Lobato, and Richard E. Turner. Stochastic Expectation Propagation. In Neural Information Processing Systems , 2015. Summary Expectation propagation (EP) is a popular technique for approximate Bayesian inference, although it has arguably lost favor in recent years to variational inference. Whereas variational inference was made scalable to massive data sets…
I’m quite excited for NIPS this year, which starts this Monday. It’s always interesting to look at the recent trends from browsing the schedule. Deep learning and reinforcement learning are of course a big highlight of NIPS, with three tutorials, a symposium, and many affiliated workshops. Approximate inference is becoming increasingly more mainstream as complex probabilistic models are fit with…
Daniel Jiwoong Im, Sungjin Ahn, Roland Memisevic, and Yoshua Bengio. Denoising Criterion for Variational Auto-Encoding Framework. arXiv preprint arXiv:1511.06406 , 2015. Summary Auto-encoders are used to learn representations of observed data . They do so by minimizing the reconstruction error of a decoder , given a characterization of hidden variables from an encoder . Along with a penalty for…
César Lincoln C. Mattos, Zhenwen Dai, Andreas Damianou, Jeremy Forth, Guilherme A. Barreto, and Neil D. Lawrence. Recurrent Gaussian Processes. arXiv preprint arXiv:1511.06644 , 2015. Summary This is another entry into the successful consolidation of Bayesian deep learning. In the Gaussian process community of recent literature, it’s been sort of a contentious issue to try to beat the practical…
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian Goodfellow. Adversarial Autoencoders. arXiv preprint arXiv:1511.05644 , 2015. Summary Following the trend in auto-encoders for generative modelling, Makhzani et al. propose an adversarial version using the framework of generative adversarial networks (Goodfellow et al., 2014). I have a bit of trouble following the terminology in these…
Eric Nalisnick and Sachin Ravi. Infinite Dimensional Word Embeddings. arXiv preprint arXiv:1511.05392 , 2015. Summary Word embeddings have been huge for the NLP community ever since Tomas Mikolov’s 2013 paper (it’s gotten over 1000 citations in 2 years!). The basic idea is to learn a -dimensional parameter vector associated to each word in a vocabulary. This is important as part of a machine…
Yishu Miao, Lei Yu, Phil Blunsom. Neural Variational Inference for Text Processing. arXiv preprint arXiv:1511.06038 , 2015. Summary Following recent work on variational auto-encoders and their advances in computer vision, the authors propose deep generative models and related inference algorithms for text. The Neural Variational Document Model (NVDM) is an instantiation of the variational…