Dyson's Bitter Lesson

I ran into an anecdote by Freeman Dyson of his encounter with Fermi1. I had read it before but I think it is interesting to cast it in the light of today’s machine learning dominated world.

• Dyson is excited about a theory on which his group has made substantial progress and seems to agree with some measurements

• The measurements happen to be performed by Fermi’s group

• Dyson meets Fermi to show their results. Fermi is skeptical with brutal honesty

• Dyson is appreciative because in hindsight, that theory turned out to be a dead-end. The answer was elsewhere. It saved him a decade of dead-end research

Buried in this are Fermi’s complaints about the pseudoscalar meson theory. He says that in theoretical physics, a calculation is on solid ground if one of the following two hold —

1. It is borne of a conceptual understanding of the problem (“clear physical picture”)

2. It is developed with a sound mathematical foundation

His primary complaint, when pressed, is that Dyson used too many parameters in his model. He responded with the classic Von Neumann retort

with four parameters I can fit an elephant, and with five I can make him wiggle his trunk.

In today’s machine learning, the predominant school of thought for “Effective Model Building” relies on

1. Data Volume

2. Model Size (# of parameters)

Some might say that this philosophy is poetically captured in Richard Sutton’s famous essay The Bitter Lesson. If you reread that essay he says

1) AI researchers have often tried to build knowledge into their agents,

2) this always helps in the short term, and is personally satisfying to the researcher, but

3) in the long run it plateaus and even inhibits further progress, and

4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.

You could say that none of this applies to science, and wave it off, fine. However, if you do interpret those statements from the perspective of a physical scientist, you might appreciate how it generates some conflict with Fermi and Von Neumann.

The Bitter Lesson argues that for his discipline, reinforcement learning, the rate at which they could make progress using just the rules (equations) they could write down turned out to be negligible in comparison to what they achieved by leveraging the sheer increase in compute capability and taking a purely data driven approach.

At first glance, it is not hard to argue that these ideas are diametrically opposed to what Von Neumann’s and Fermi’s ideals2. One could interpret Sutton as saying “you can’t come up with and write equations fast enough, you should just use a super flexible model.”3

But I think the Bitter Lesson is a higher level claim

I think the more salient point to physical scientists is this line —

One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great.

For them, those methods that scaled well were “search and learning”. I think it is important to ask yourself, as a computational scientist, what that means to you.

But it is paramount, in my opinion, to start scaling the computational complexity of your work. The available computation is quite large, and it behooves you, fellow scientist, to use it, and use it in different ways.

• Automate your data processes

  • analysis <> stop analyzing a simulation individually

  • visualization <> stop analyzing a simulation individually!

  • storage <> don’t spend time moving folders around

  • organization <> definitely don’t spend time moving folders around and then looking for them later!

• Scale your compute

  • run many simulations <> stop running single simulations

  • run different kinds of simulations <> stop working with that same equation

  • use GPUs (if you can)

• Change your compute

  • Get and use gradients

  • Do outer loop science (optimization, inverse design, uncertainty quantification etc.)

  • (Judiciously) leverage machine learning

Thanks for reading!

It doesn’t really, but you could ask what Von Neumann and Fermi would think of differentiable simulations given the statements above

Well, we’re further than we would like from Von Neumann’s ideals. My puny brain probably can’t even imagine what he could do with a billion parameter neural network…

BUT, we are cooking with Fermi’s 2 points. We do base our modeling on physical intuition, and we do build on the sound mathematical foundation provided by the partial differential equations. So that’s something, and that’s kind of the point I tried to make in my previous post