Recent posts from Conversations
Why is playing chess fun? Why are people irrational? What is randomness? Why does cryptography work? What do these questions have to do with each other? Let’s start with chess. Can you imagine a perfectly rational agent, a Homo Economicus , if you will, playing chess? To be perfectly rational is to know all the consequences of everything one knows. Knowing the rules of chess, our Homo Economicus…
Infinity. What is it, really? Perhaps you remember, as I do, the moment in your life when you realized one could (in principle) count forever, and there would always be integers there for one to count. Yet, how can our finite minds comprehend the truly infinite? Do infinte things even exist? My own thoughts on this subject have gone full circle. I accepted the infinitude of the integers at a young…
Previously in this miniseries on testing samplers, I laid out the problem that statistical testing is inherently more fraught than conventional software testing, and set up a basic framework for defining dependable error-calibrated statistical tests of stochastic software. I have since learned a great inequality, and can now lay out a more complete set of practical basic tests. To recap, the…
How do you unit-test a sampler? Run it a bunch and see whether the histogram looks good—but there’s always some chance that your test will fail randomly even though the sampler is good, or pass randomly despite a bug. And if you try to have more than one test, the chance of random errors goes up. How big of a chance? How much worse does it get? What to do? I’ve written about this problem before.…
People in my circles periodically talk and write about the nature of this emerging new beast that is called probabilistic programming. There’s various talk about how it’s about samplers, or density functions, or Markov chains, or making machine learning or artificial intelligence more accessible, or various other such stuff. Certainly those things hover in the air around probabilistic programming,…
Testing traditional software strives for an ideal of exact determinacy: If a test fails, there is a bug and the developers must investigate; and if a test passes, the bugs it is testing for are not present, so development can proceed without worrying about them. This is not possible with probabilistic programs —even a correct sampler for a probability distribution always has some chance of…
Programming intentionally random programs presents its own special software engineering considerations, in addition to the usual ones. I have been surprisingly slow to realize this, but two years in to working on a probabilistic programming platform I can recommend some specific habits around software engineering of intentionally random programs. Reproducibility . Ironic as this may sound, my…
For a little over two years, I have been professionally dealing with programs whose behavior is intentionally random. Why would one even have intentionally random programs? The first kind of “random” program I encountered was the randomized algorithm , but I do not consider these to be intentially random in this sense. Such a program’s purpose is to compute something deterministic, and randomness…
Good software is software that admits a simple mental model. For all that I have observed and participated in plenty of discussions about one or another piece of software as to whether it is or is not good, I am surprised to say that I have never seen a single defining principle clearly articulated. Permit me, therefore, to propose this one: Good software is software that admits a simple mental…
Possibly the greatest tension in the design of programming languages occurs when encountering a user program that doesn’t quite make sense. The two coherent schools of thought on the subject are The platform should make its best effort to continue the computation—choose reasonable rules for resolving potential ambiguities, so as to avoid crashing the program until absolutely necessary. The…
What makes a good representation for computing with probability distributions? The two canonical options are samplers and probability density functions. Both are valuable; and the relationship between them turns out to hide two fruitful variations on the idea of a sampler, that I will call “importanter” and “rejecter”. The purpose of this essay is to carefully study these four objects and the…
Observing the state of modern software practice, one might notice that a rough tower of different abstraction mechanisms has emerged. One might then wonder: why do we need so many concepts, if their fundamental job is to give a name to some software, and allow one to use it just by referring to its name? Indeed, why do we need functions (procedures, methods), and classes (in languages that have…
In English, quantifiers are words like “all”, “one”, or “some” that indicate how broadly true the quantified clause is. Formal logic has adopted symbols for such words, namely “all”, “exactly one”, and “some” (in the sense of “at least one”). Probability theory offers us a reason to incorporate another symbol, with a meaning along the lines of “some, and I have a sense of how to find them”. I want…
This essay is inspired by the assertion that hard work is more important for success than being smart. There are, of course, many meta-objections one could have against such an assertion, especially in the context in which I heard it. For example, one could notice that the speaker, being American, might be culturally obligated to say something like this. Or one could observe that speaking to an…
Test Driven Development, as it is now called, is something I have advocated for a large part of my career. My more recent experience, however, has brought more nuance to my view of the proper level and kind of software testing, which now feels ripe for setting down. The driving observation is that automated tests bear a cost. Tests are code, and like all code they require maintenance. In the case…
Advertising. People in my social circle are used to thinking of it as being an awful institution: greedy, faceless, unscrupulous corporations brainwashing the susceptible masses into wasting their hard-earned money on cheap, low quality, superfluous devices for solving problems no one actually has and that don’t even work. An irritation at best, sheer deceit and manipulation at worst. But in fact,…
Computationally generating the semantics of a web page—the HTML—has become the standard modus operandi on the web. No serious web developer writes by hand the HTML that their server sends to the browser anymore; invariably, some template engine or other generator intermediates, removing tedium and adding flexibility and power. The same thing is slowly happening to the presentation layer as…
The Cleverness of Compilers essay described the name of the hyperaggressive compilation game in broad, philosophical strokes. Here, I would like to walk through the Mandelbrot example in some detail, so that the interested reader may see one particular way to actually accomplish that level of optimization. There are of course other approaches to the same goal, and the present approach (as well as…
The “Sufficiently Clever Compiler” has become something of a trope in the Lisp community: the mythical beast that promises language and interface designers near-unlimited freedom, and leaves their output in a performance lurch by its non-appearance. A few years ago, I was young enough to join a research project to build one of these things. Neglecting a raft of asterisks, footnotes, and caveats,…
Automatic differentiation may be one of the best scientific computing techniques you’ve never heard of. If you work with computers and real numbers at the same time, I think you stand to benefit from at least a basic understanding of AD, which I hope this article will provide; and even if you are a veteran automatic differentiator, perhaps you might enjoy my take on it. What it is Wikipedia said…
The successful fox must know more than the sum of what the hedgehogs know, for it must know the connections from one thing to another. This fact is key to the design of computer systems for solving certain kinds of problems. I keep coming across domains with the following structure: One wishes to solve a large, complex problem that fits into some uniform formalism (for example, a detailed…
If programming languages were building materials, Python would be mud. It’s everywhere. If you need to slap something together, it’s easy—just wash your hands afterwards. If you need to change it, you just push on it, and it deforms around you. But if you push it too far, it falls over, and if you leave it out in the sun for a few days, it dries out and turns brittle. If programming languages were…
This Tuesday Belmont, MA, the town where I live, makes a contentious decision: To vote for or against an increase in property taxes. It’s a contested election: I’ve received three flyers in favor and four against, to say nothing of comments online and by folks I meet around the town. I want to work out a good way to think about this kind of choice. The Mechanics Belmont is in Massachusetts, and…
A friend recently pointed me to an article arguing that the Clean Development Mechanism may be counter-productive if it mis-estimates how additional its supported projects are, and ends up allowing more emissions elsewhere than it actually reduces. Does this mean offsets are useless? As an individual, does buying and retiring offsets still hold water as a climate charity? Let’s work through a…
Over the years, I have developed a fairly strong theory of how to organize, and what the end-goal of organizing is. Since people occasionally ask me, I figured I would write it down. Of course, articulating also helps me refine my own understanding. Without further ado, Organizing a space has two goals: To make it easy to pick up and put everything away. To make it easy to get whatever is needed.…
Every once in a while someone hears me use the phrase “yak shaving” to describe some activity and asks what I mean. Here is a superb example from my recent life: I want to lower the interest rate on my mortgage. So I start an application to refinance. During the application, I discover that my credit score is lower than I expect. So I pull my credit reports, and find an error. In order to dispute…
Production-level probabilistic inference is usually said to be about very high-dimensional problems. The usual argument for the techniques one learns (importance sampling, Markov chain Monte Carlo, etc) starts from the curse of dimensionality—that classic quadrature is hopelessly inefficient in many (i.e., more than four) dimensions. But what if one wants probability in a low-dimensional problem?…