The burdens of optimization

Optimizing is everywhere. Oura rings promise optimal sleep. Consultants tout optimized operations. Startups even help Silicon Valley parents select embryos that are “optimized for IQ.”

But I think we need to be careful about optimizing. It takes a lot of work to do it right, and sometimes it’s completely unnecessary—a lesson I’ve learned the hard way. I’ve made enough mistakes with optimizing to learn that there are three questions to answer before jumping into it:

1. Am I optimizing for the thing I actually care about?

2. If so, is it even worth it to optimize that thing? (Or is “good enough” ok for now?)

3. If so, am I optimizing it the right amount? (Or am I over-optimizing?)

A few minutes on these questions can save you weeks of optimizing the wrong thing, too early, and too much.

Am I optimizing for the thing I actually care about?

“Optimize” is such an authoritative word that it’s easy to assume “optimizing something” is the same as “accomplishing the goal I care about.” But notice that these two things can be easily confused:

1. Optimizing sales is not equivalent to optimizing profits.

2. Optimizing money is not equivalent to optimizing happiness.

3. Optimizing follower count is not equivalent to optimizing for a viable business.

In The Art of Doing Science and Engineering, Richard Hamming pointed out that optimizing for the right thing can get especially fiddly when you’re working in a system with a lot of moving pieces. If you’re in charge of just one piece, optimizing it may seem like an obvious choice, but it can cause the overall system to perform worse. For example, if you’re designing the engine of a car and you optimize the engine for horsepower, the car’s engine will be too heavy. Optimal engine, suboptimal car.

So don’t forget what you’re really optimizing for! And if you’re in charge of a smaller thing within a bigger team, make sure you’re optimizing for the larger goal that your team cares about, not just the little thing you’re in charge of.

Is it even worth it to optimize? (Or is “good enough” ok for now?)

Optimizing something is fun and interesting because it gives you a specific, clear challenge and an actionable to-do list for the day. That’s also why it’s dangerous. It’s easy to fall into premature optimization, the mistake of optimizing the right thing too early:

1. You optimize your backend when you’re not sure customers even want your product.

2. You optimize a customer’s operations when the customer is still struggling with fundamental operational challenges you could solve first (I’ve made this mistake).

3. You optimize your landing page when you’re two weeks into creating something new.

Optimizing an existing thing will always feel easier than trying and failing at a new thing. So be careful about optimizing something when you should be moving on to something else. Remember the 80/20 rule: 80% of impact comes from the first 20% of effort.

Am I optimizing it the right amount? (Or am I over-optimizing?)

You’ve picked the right thing to optimize, and you’ve decided that it’s indeed worth the extra time and effort to optimize it. The final step is to make sure you actually optimize it to the right degree.

It’s really easy to get greedy here and over-optimize. You’re so eager to squeeze out every little inefficiency that you don’t leave any room for error. Then when the teeny-tiniest hiccup or delay happens, things completely fall apart. (And then the media, dripping with schadenfreude, makes fun of you for trying to over-optimize.) Example: an airline keeps only the “optimal” number of planes for forecasted demand at its airports. Then, a single cancellation in Worcester snowballs into a system-wide outage.

Over-optimization is also problematic when consultants or “value creation” people enter a new business and start hacking away at “inefficiencies” to “optimize operations.” The problem is that sometimes what appear to be inefficiencies are actually buffer (or slack) that add resilience to the overall operations. Another example: when the pandemic scrambled supply chains, companies that had optimized inventory down to nearly zero (“just-in-time” inventory) suddenly couldn't make anything.

When you optimize the right metric at the right time, don’t get too greedy. Make sure you leave some buffer room for things you can’t account for. This might mean you stay a bit “under” the optimal point, which might feel like you’re wasting resources. But you’re trading short-term efficiency for long-term resiliency.

The burdens of optimization

Something I learned writing this: the word “optimize” didn’t enter the popular lexicon until the 1950s—when “computing machines” finally made real optimization possible. Before that, people didn’t worry as much about optimizing, because there wasn’t a word for it. They just did the best they could.

We now live in a different world. 200-IQ AI is getting cheaper every day, and we have more data on stuff than ever before. We can use both to optimize portfolios, hiring, and sleep scores. It’s easy to feel a burden to optimize everything in the world...because you can.

But you shouldn’t. Optimizing is a waste of time when you optimize for the wrong thing, too early, too much. You work so hard optimizing, thrashing about and writing ludicrous formulas in Excel, only to find that you could have accomplished more doing nothing (or reading Don Quixote).

So I’m trying to let go of the burden to optimize everything, especially in my personal life. I no longer beat myself up if I don’t get exactly eight hours of sleep or eat exactly the right amount of protein or work on something for exactly as long as I hoped. Smartwatches, rings, and bracelets love to incentivize over-optimization, because it correlates with “user engagement.” Oh no, your sleep score was only 70 because you chose to meet a friend for dinner! Bad, Brendan! It may be possible to be “optimal,” but isn’t it much more fun—and sustainable—to be a little ~gasp~ suboptimal? I like Scott Galloway’s take on this:

“I’m health conscious 80% of the time, so I can devour the other 20%, and create a whole that is greater than the sum of its parts. The question isn’t will I live longer, but will I live better? A: Yes.”

It’s liberating to be relieved of the burdens of optimization. You prioritize the right things, accept that good enough is often better than perfect, and cut yourself some slack so you don’t need to operate at 100% just to avoid catastrophe.

Further Reading

1. The Art of Doing Science and Engineering, Richard Hamming

2. Thinking in Systems, Donella Meadows