Age, Uncertainty, and Draft Decisions

This wasn’t the plan.

With the NFL draft around the corner, I intended to build a simple model and write about my projections for this year’s crop of quarterbacks. But after fiddling with inputs for a few days, my results remained no better than simply using draft position.

Fortunately, not all was lost. In my attempts to improve performance, one variable continually showed signal: age at the time of the draft.1 And that’s a more interesting subject to talk about.

At this point, most, if not all, professional organizations use a model to help inform draft decisions. The type of model, and the degree to which it’s relied upon may vary, but they have something that aggregates information to predict how good players will be.

And if my experience is at all representative, the inclusion of age in those models elicits strong feelings.2 That’s because 1) it clearly adds predictive value above and beyond scouting reports and performance and 2) where that value comes from is misunderstood.

When most people hear that age is important for draft picks, they assume it means:

• Younger players have more time remaining for physical development

• Younger players have played less and therefore their skills will develop quicker with professional instruction

• Younger players are more athletic with higher upsides

Despite being plausible, none of these reasons explain why age matters. Or maybe they do. Because we don’t know, with any real certainty, why age shows signal. All we know is that the combination of scouts and statistics alone under-project young players and over-project old players. Anything beyond that is conjecture.

That’s a hard truth to accept. Uncertainty is far less comforting than an understood narrative.

And if it stopped at narrative, it wouldn’t be a problem. Unfortunately, it rarely stops at the rationale; that rationale is then weaponized to dismiss age when the story doesn’t fit. Anyone who’s been in draft meetings can attest. The young player who is physically mature moves down the board, or the older player who only recently began playing their sport rises.

It could be correct to discount age in those cases, but it’s not based on evidence. The objective information says that predictions improve when they account for age.

That leads to the important question: how should teams incorporate age into their models?

Because, like most things, it’s not as simple as taking the output at face value.

Draft models are built on history. A good training set — something recent enough to be useful but with enough time to understand outcomes — will consist mostly of data that’s 10 or more years old.3

But things have changed dramatically, for any organization, over ten years. Many of the scouts who wrote the old reports have moved on to new teams. Player development has modernized with new technologies and added emphasis on training, nutrition, and recovery. And maybe most importantly, teams understand far better that age matters.

Since Rany Jazayerli’s groundbreaking article 15 years ago, scouts have slowly acknowledged that young players tend to perform better.4

If evaluators are aware that age matters, and that awareness influences their reports, including a model adjustment for age runs the risk of overcorrecting. Then again, there’s plenty of research to suggest that awareness of bias isn’t enough to overcome bias. Even if scouts are factoring in age more than they did previously, the signal may be as strong as ever.

Teams use models because they’re better than humans at combining and weighing information, which is really what decisions in the draft are all about. But there’s a reason the most well-known quote about models tells us that only some of them are useful. Models don’t (perfectly) reflect reality and they require a great deal of choice and understanding to use effectively.

You could convince me that nearly any magnitude of adjustment for age makes sense; you just won’t be able to prove it.

The inclusion of age boils down to belief. Based on everything — history, environment, people — what do we believe is appropriate? What bet are we comfortable making?

Fernando Mendoza, the presumptive top pick in next Thursday’s NFL draft, is 22.5 years old. That’s slightly younger than the average quarterback drafted this century. How much that matters is a question that every NFL team has to answer for themselves.