Measuring 500 Nuclear Engineers

One signal 🔭
One subtraction ➖
One analogy 📟

Created by Sam Rogers | Episode 7 with Douglas Hubbard | Issue 64 of Sunday newsletter of the weekly show at sigsub.show | Originally published at https://sigsub.show/newsletter/measuring-500-nuclear-engineers/

Douglas Hubbard has spent a career being told things cannot be measured. He says there are only three reasons anyone ever believed that, and all three are illusions: concept, object and method.

Concept is the belief that a measurement is one exact number. In the empirical sciences it never is. A measurement is a range that got narrower. Hold out for a single true figure for what AI returned and you will wait forever, or take the first confident number somebody hands you.

Object is the thing being measured never actually getting defined. Productivity. Collaboration. Enablement. Ask what you would see more of if you had more of it, and the room goes quiet.

Method is assuming you need the whole population before you are allowed to begin. You don’t. You need a sample and a threshold.

He described all three on a podcast with me a decade ago, before there was an AI budget anyone had to defend. Nothing in it needed updating, which is the uncomfortable part.

These are not new failures. They just have bigger budgets attached now, and a date by which somebody has to say something.

Share

Hubbard kills a bad measurement with one question: why do you care? Not what will you learn. What will you do differently.

Most AI pilots cannot answer it. They produce a percentage, a satisfaction score, a time-saved figure, and then everybody proceeds exactly as they would have. By his definition, that’s not measurement.

The diagnostic: the flip test. Before the pilot starts, write down the decision it feeds and the number at which you would do the opposite. Roll it out above X. Kill it below Y. If you cannot name X and Y, no result changes anything, and you can cancel it this morning and lose nothing but the extra meeting invites.

Hubbard’s version is the same point with math behind it: if your uncertainty straddles a threshold that matters, measuring is worth it. If it doesn’t, it isn’t.

Episode 7 is audio only, and a little different from the usual: a 2016 interview from my own archive, cut with a present-day introduction and close.

Listen at Substack, or find every format in one place at sigsub.show/episodes/ep-007. Also on Apple Podcasts, Spotify and Pocket Casts, wherever you follow the show, and cross-posted to the LinkedIn newsletter.

What to listen for: Hubbard asks me how I would measure collaboration, I give the answer everybody gives, and he takes it apart. That exchange is why this tape came out of the archive rather than staying in it.

Long before anyone had DMS dashboard, Hubbard was asked how much time nuclear engineers spent on document handling. Five hundred of them, scattered across different locations, doing different work on different days.

He didn’t survey them. He didn’t install anything on their devices. He messaged each person exactly once over the course of a month.

When the notification came, the engineer wrote down what they happened to be doing at that moment. On the phone. In a meeting room. On this project, doing that task.

500 moments from 500 people, spread across different hours and different weeks. Nearly 100 turned out to be work a document management system would have done for them. And if the message caught somebody in the break room, they simply documented “break room” which doesn’t mean they spent all day in the break room. The spot check didn’t say anything about the individual, and it didn’t need to.

Nobody followed a single engineer for a single day. That is how they found out what five hundred of them did with a month.

Concept, object, method. Three illusions, and the same three sit underneath every AI value case that will not resolve.

So run the flip test this week, on whichever pilot is closest to a real decision. Name the decision. Name the number that flips it. If no number does, you already have your answer, and it cost you nothing to find out.

Then the part Hubbard would want said last: you almost certainly have more data than you think, and you need far less of it than you think. If you know almost nothing, almost anything tells you something. The first few observations do the heaviest lifting they will ever do, which means the cheapest measurement available to you is the one you have not started yet.

Good luck,

Sam Rogers
Messaged Once, Politely

New here? Subscribe at sigsub.show and the next episode lands in your inbox.

The show goes live again Wednesday, and cohost JD Dillon is bringing a signal that rhymes with this one harder than either of us planned.

I’d rather influence 10 people who can actually change something than reach 10,000 people who scroll past it.

JD Dillon

Which is this week’s argument wearing different clothes. Hubbard’s point is that you need far fewer observations than you think. JD’s is that you need far fewer people. Both are the same refusal to confuse volume with evidence, and you will hear them collide on Wednesday.

• Confidence & Calibration (issue 023). This newsletter covered Hubbard’s calibration research back in Volume I and said the interview still held up. Episode 7 is the show acting on its own advice.

• On Time or Ready? (issue 052). Pressure does not make anybody better at measuring. It makes them faster at producing a number, which is the one Hubbard spends this episode talking you out of.

• The Bull in the Apartment (issue 061). Paperwork that answers procurement’s question without ever showing that the value survived. A pilot that flips no decision is the same move in a smaller room.