In the first post of this series, I have outlined the importance of having a proper operational framework in place before starting any talk of information-theoretic quantities. It is, however, a good idea to pin down even provisionally the kind of information-theoretic quantity one hopes to distill out of the operational formulation. This will be Continue reading "Information and Control in…
Disclaimer: I am not a biologist, but I have become interested in biology and related matters over the past couple of years. One reason is obviously the pandemic, so the talk of biology, viruses, mRNA, and the like is everywhere. The other, main, reason is that I think we will not get anywhere interesting in Continue reading "Information and Control in Biology, Part 1: Preliminary Considerations"
These notes are based on the tutorial I gave at the Geometric Methods in Optimization and Sampling Boot Camp at the Simons Institute in Berkeley. Suppose we wish to obtain samples from some probability measure on . If has a sufficiently well-behaved density with respect to the Lebesgue measure, i.e., , then we can use Continue reading "Sampling Using Diffusion Processes, from Langevin to…
Machine learning is about enabling computers to improve their performance on a given task as they get more data. Can we express this intuition quantitatively using information-theoretic techniques? In this post, I will discuss a classic paper by David Haussler, Michael Kearns, and Robert Schapire that (to the best of my knowledge) took the first Continue reading "Counting bits with Vapnik and…
Models of complex systems built from simple, locally interacting components arise in many fields, including statistical physics, biology, artificial intelligence, communication networks, etc. The quest to understand and to quantify the fundamental limits on the ability of such systems to store and process information has led to a variety of interesting and insightful results that Continue reading…
1. The 2014 IEEE North American Summer School on Information Theory will take place June 18-21, 2014 at the Fields Institute in Toronto, Canada. 2. For those of you who use Matlab or Octave, there is a new Information Theoretical Estimators (ITE) toolbox, an open-source toolbox capable of estimating many different variants of entropy, mutual Continue reading "Two public service announcements"
Just a quick note for my reference, but it may be of interest to others. Let denote the size of the largest code over a -ary alphabet that has blocklength and minimum distance . The well-known Gilbert-Varshamov bound says that where is the volume of a Hamming ball of radius in . The usual way Continue reading "A graph-theoretic derivation of the Gilbert-Varshamov bound"
circa 1935. Here is a quote from Single-Story America (translated as Little Golden America), an American travelogue by Ilya Ilf and Yevgeny Petrov: There is a whole book of readymade telegrams, long and convincing, lavishly composed telegrams for all occasions. Sending such a telegram costs only twenty-five cents. You see, what gets transmitted over Continue reading "Lossless source coding at…
Of the five plenary talks at this year s ISIT, two were about concentration of measure: Katalin Marton s Shannon lecture on Distance-divergence inequalities and Gabor Lugosi s talk on Concentration inequalities and the entropy method the next morning. Since the topic of measure concentration is dear to my heart, I thought I would write down a few Continue reading "ISIT 2013: two plenaries on…