Machine Learning and Statistics
Efficiently estimating the trace \(\textrm{tr}(A) = \sum_{i=1}^d A_{ii}\) of a square matrix \(A \in \mathbb{R}^{d \times d}\) is an important problem required in a number of recent deep learning and machine learning models. In those cases the matrix \(A\) is typically positive-definite , large …
TLDR : in the past decade longevity has emerged as a serious research field. There are now a number of studies that indicate that a number of safe supplements may likely extend lifespan and health in adult humans. Note : I normally blog about statistics and machine learning. This article is different …
This year at NeurIPS 2018 the Symposium on Advances in Approximate Bayesian Inference discussed challenges and advances in approximating probabilistic inference in rich models. It was a genuinely exciting program! I was lucky enough to give an invited talk at the event. Title: Debiasing Approximate Inference Abstract: At its heart …
The Machine Learning Summer Schools (MLSS) is the largest and most popular machine learning summer school series. For two weeks in August and September the MLSS 2018 is held in Madrid . I am happy to speak on the topics of generative adversarial networks (GANs) this year. My talk materials are …
TLDR: Yes, and there are precise results, although they are not as well known as they perhaps should be. Over the last few years I had many conversations in which the statement was made that Bayesians methods are generally immune to overfitting, or at least, robust against overfitting, or---everybody would …
We just published an article discussing our recent work on stabilizing generative adversarial networks .
The biggest AI conference of the year has just ended: NIPS in Barcelona broke all records this year and the program was exciting as always. It certainly remains my favorite conference to attend. One of the best things about NIPS are the numerous high-quality workshops; this year David Lopez-Paz , Alex …
A new book, Computer Age Statistical Inference: Algorithms, Evidence, and Data Science by Bradley Efron and Trevor Hastie , was released in July this year. I finished reading it a few weeks ago and this is a short review from the point of view of a machine learning researcher. Living in …
A common numerical operation in statistical computing is to compute $$\log \sum_{i=1}^n \exp x_i,$$ where \(x_i \in \mathbb{R}\) , and \(n\) is potentially very large. We can implement the above computation by exponentiating each number, then summing them, then taking a logarithm as follows (written in Julia …
Artificial Intelligence (AI) is making progress in great strides, or at least it appears so! Almost no week passes by without some major announcements of new challenges solved by AI technology or new products powered by AI. Indeed many quantifiable factors attest an unprecedented level of activity: capital investments, number …
Nowadays authors in the fields of statistics and machine learning often choose to write their books openly by publishing early draft versions. For popular books this creates a lot of feedback and in the end clearly improves the final book when it is published. Here is a short list of …
(This article covers the decision-theoretic concept of value of information through a classic example.) What is the value of a piece of information? It depends. Two factors determine the value of information: first, whether the information is new to you; second, whether the information causes you to change your decisions …
This article summarizes the fourth day of the ICCV 2015 conference, the International Conference on Computer Vision. A summary of the first day , second day , and third day is also available. ICCV 2017 and 2019 ICCV 2017 will be in Venice, Italy. For ICCV 2019 there was an open voting …
This article summarizes the third day of the ICCV 2015 conference, the International Conference on Computer Vision. A summary of the first day and second day is also available. Interesting Papers Registering Images to Untextured Geometry Using Average Shading Gradients By Tobias Ploetz and Stefan Roth. This work considers the …
This article summarizes the second day of the ICCV 2015 conference, the International Conference on Computer Vision. A summary of the first day is also available. Awards The following awards were given at ICCV 2015. Achievement awards PAMI Distinguished Researcher Award (1): Yann LeCun PAMI Distinguished Researcher Award (2): David …
ICCV 2015 , the International Conference on Computer Vision, is one of the premier venues for computer vision research, together with the CVPR conference. This ICCV is happening in Santiago, Chile, a beautiful city with amazing food. The computer vision community is growing, and this ICCV is the largest so far …
This continues the first part on tips to write computer science papers. 6. Ideal Structure of a Paragraph A paper has different levels of formal structure: sections, subsections, paragraphs, sentences. It is important to ensure that the structure of the content aligns well with the formal structure because the formal …
As a non-native English speaker I can relate to the challenge of writing concise and clear English. Scientific writing is particularly challenging because the audience is only partially known at the time of writing: at best, the paper will still be read in 10 or 20 years from the time …
This is the third part of a three part post. The first part covered the early history of Monte Carlo and the rejection sampling method, the second part covered sequential Monte Carlo. Part 3 In this part we are going to look at Markov chain Monte Carlo. The video files …
This is the second part of a three part post. The last part covered the early history of Monte Carlo and the rejection sampling method. Part 2 In this part we are going to look at importance sampling and sequential Monte Carlo. The video files are also available for offline …
Some time ago in June 2013 I gave a lab tutorial on Monte Carlo methods at Microsoft Research. These tutorials are seminar-talk length (45 minutes) but are supposed to be light, accessible to a general computer science audience, and fun. In this tutorial I explain and illustrate a number of …
Julia is a relatively new programming language with the declared goal to become the leading language for scientific computing. I have probably annoyed half of my colleagues by raving about how great the language is and what it is good at. Before we get to this, and in my defense …
This post continues the previous post, part 1 on scoring rules. However, today we will be more hands on, testing your skill of making good and well-calibrated predictions. To this end, I will ask you several questions about numerical quantities and I would like to hear an answer stated as …
This article is the first of two on proper scoring rules , a specific type of loss function defined on probability distributions or functions of probability distributions. If this article sparks your interest, I recommend the gentle introduction to scoring rules in the context of decision theory in Chapter 10 of …
Michael Hirsch and myself are organizing a workshop on the topic of machine learning for image and video processing as part of the ICCV 2015 programme . The workshop takes place on the 17th December 2015 in Santiago, Chile, right after the main ICCV conference. Call for Contributions Image processing methods …
In this article we will look at a practically important measure of efficiency in importance sampling, the so called effective sample size (ESS) estimate. This measure was proposed by Augustine Kong in 1992 in a technical report which until recently has been difficult to locate online, but after getting in …
One of my all-time favorite algorithms is reverse search proposed by David Avis and Komei Fukuda in 1992, PDF . Reverse search is an algorithm to solve enumeration problems, that is, problems where you would like to list a finite set of typically combinatorially related elements. Reverse search is not quite …
This post is about a recent arXiv submission entitled Gradient Estimation Using Stochastic Computation Graphs , and authored by John Schulman , Nicolas Heess, Theophane Weber , and Pieter Abbeel . In a nutshell this paper generalizes the backpropagation algorithm to allow differentiation through expectations , that is, to compute unbiased estimates of $$\frac{\partial …
This article is about multilevel splitting , a method for estimating the probability of rare events. Estimating the probability of rare events is important in many fields. One vivid example is in the study of reliability of systems; imagine for example, that we are responsible for building a mechanical structure such …
P-Values (see also Jim Berger's page on p-values ) are probably one of the most misunderstood concepts in statistics and certainly have been abused in statistical practice. Originally proposed as an informal diagnostic by Ronald Fisher , there are many reasons for the bad reputation of p-values, and in many relevant situations …
The multivariate normal distribution is one of the most important probability distributions for multivariate data. In this post we will look at the entropy of this distribution and how to estimate the entropy given an iid sample. For a multivariate normal distribution in \(k\) dimensions in standard form with mean …
CVPR 2015 , "Computer Vision and Pattern Recognition" is the main conference of the computer vision community and just finished. I unfortunately was only able to stay for the three main conference days, but here is my short subjective summary. For an overview of individual research papers, see this excellent summary …
This article describes the basic problem of image demosaicing and a recent work of mine providing a research dataset for demosaicing research. Image demosaicing is a procedure used in almost all digital cameras. From your smartphone camera to the top-of-the-line digital SLR cameras, they all use a demosaicing algorithm to …
This post continues the previous post, part 1 and part 2 , outlining my criticism towards a ''naive'' subjective Bayesian viewpoint: The consequences of model misspecification . The ''model first computation last'' approach . Denial of methods of classical statistics, in this post. Denial of the Value of Classical Statistics Suppose for the …
This post continues the previous post, part 1 , outlining my criticism towards a ''naive'' subjective Bayesian viewpoint: The consequences of model misspecification . The ''model first computation last'' approach, in this post. Denial of methods of classical statistics . The ''Model First Computation Last'' approach Without a model (not necessarily probabilistic) we …
I have used probabilistic models for a number of years now and over this time I have used different paradigms to build my models, to estimate them from data, and to perform inference and predictions. Overall I have slowly become a Bayesian; however, it has been a rough journey. When …
An amazing fact in high dimensions is this: Projecting a simple convex set described by a small number of inequalities can create complicated convex set with an exponential number of inequalities. It is amazing because it contradicts our everyday human experience. We are most familiar with projections of objects in …
Error bars and the \(\pm\) -notation are used to quantitatively convey uncertainty in experimental results. For example, you would often read statements like \(140.7 \textrm{Hz} \pm 2.8 \textrm{ Hz SEM}\) in a paper to report both an experimental average and its uncertainty. Unfortunately, in many fields (such …
In the last two parts ( part one , part two ) we looked at the problem of entropy estimation and several popular estimators. In this final article we will take a look at two Bayesian approaches to the problem. Bayesian Estimator due to Wolpert and Wolf The first Bayesian approach to entropy …
This year we ( Zoubin , together with David and myself) are again organizing a workshop event for the local Cambridge (UK) machine learning community. The schedule is available at the workshop homepage, Machine Learning in Cambridge 2015 , and we also plan to make all talks available as video recordings after the …
In the last part we have looked at the basic problem of discrete entropy estimation. In this article we will see a number of proposals of improved estimators. Miller Correction In 1955 George Miller proposed a simple correction to the naive plugin estimator \(\hat{H}_N\) by adding the constant …
Estimation of the entropy of a random variable is an important problem that has many applications. If you can estimate entropy accurately, you can also estimate mutual information , which allows you to find dependent random variables in large data sets. There are numerous applications . The setting of discrete entropy estimation …
In December 2014, just in time for NIPS, MIT Press released an edited volume on structured prediction models and their applications in natural language processing, computer vision, and computational biology. Advanced Structured Prediction , Editors Sebastian Nowozin, Peter V. Gehler, Jeremy Jancsary, Christoph H. Lampert, ( MIT Press , Amazon ) The volume offers …
Given a sequence of observed data we would often like to estimate simple quantities like the mean and variance. Sometimes the data is available in a streaming setting, that is, we are given one sample at a time. For example, this is the case when the number of samples is …
This is the start of my blog. This will be a quite technical blog and therefore address a more specialized audience. The articles will cover topics in the area of machine learning, statistics, maybe some computer vision, let's see. I plan to publish one article every two weeks, but let …