Mailbag: What isn't measurable? To hire as exec or not?

This is a mailbag of several different questions I’ve been asked over the past few weeks.

What isn’t measurable?

A friend of mine is interviewing for engineering leadership roles and brought an interesting question into our group chat from one of their interviews, “What are outcomes that you can’t measure with metrics?” For the record, I’m skeptical this is a good interview question, my most generous read is that it’s trying to detect pragmatic, experienced leadership because experienced leaders inevitably should have encountered things that matter but are hard to measure. I’d also presume that the follow up question is pushing the candidate to offer an approach to trying to measure the immeasurable topic.

My general thinking rules here are:

1. Some present values can be measured directly, e.g. uptime, time to close tickets over past week
2. All present values can be measured indirectly through a proxy measurement (see also Forecasting synthetic metrics) , e.g. team morale can be approximated through employee retention, product quality can be approximated by number of support tickets
3. You can forecast the future value of any present value, e.g. future time to close tickets, future employee retention

That said, if the goal of this kind of interview question is to have a discussion that simulates working together, then maybe a more interesting answer to this question is simply, “engineering velocity.” Almost every senior engineering leader is asked to measure engineering velocity on their team, and all answers to this question are deeply unsatisfying: features shipped, tickets closed, etc.

At Calm, I’ve focused on measuring engineering through the number of features shipped, maintaining an agreed upon target win/loss/neutral rate for experiments (e.g. roughly one-third in each bucket), and whether we made exactly one big technical investment per quarter. For many folks asking for an engineering velocity measurement, this will meet their criteria, but it’s really more of an investment thesis. Even if we shipped more features one quarter than another, I wouldn’t actually believe that our velocity had necessarily gone up, it’s more likely that the features themselves were smaller.

I generally think engineering velocity measures are only interesting for debugging a low velocity team, rather than particularly helpful in ensuring a team is high impact. What we really want to be measuring is “engineering impact.” I asked about measuring engineering impact on Twitter a while ago, and the answers were not too helpful. Generally folks recommend measuring engineering impact through the lens of business impact, which makes a lot of sense, but is a bit “measuring the output rather than the input” in my opinion. This certainly helps you grade the team’s impact, but it does less to help you become more impactful.

Should we hire this candidate as an executive?

A founder at a Series A startup asked me whether they should hire a product leader as an executive (e.g. VP Product) or should hire them into a less senior role (e.g. Director of Product). Interviewers were split on which was the right decision, but it was also a strong candidate that they didn’t want to lose.

Honestly, I don’t think I have the right answer here, but my default mental model here for hiring an executive at an early stage startup is roughly:

1. Do they bring domain expertise to your field or are they uniquely skilled at an area that is essential for your success? Later stage executives need a broader skill set and consequently don’t have to go as deep on any given area (their teams can compensate for those gaps), but earlier stage executives need to bring some aspect of truly exceptional strength; hiring a generally good, later stage executive into an early stage role is rarely a good outcome for either the executive or the early-stage company
2. Can they both do the team’s work themselves and scale to providing the structure/strategy for a team of folks to do the work beneath them in org structure? If not, they are likely to be the wrong person in the role if you are successful
3. Have they managed and hired managers before? If not, they will have a challenging time growing their team if you are successful

If they meet those three criteria, then it makes sense for an early startup to hire them as an executive. If they don’t, I’d hire them in a less senior role to watch them work for a while. That said, the details will always depend on your specific company and the specific candidate. Most executives won’t land the transition from Series A to Series D, and a self-aware executive is self-aware of that! The risk of adding them as an executive is much lower if they’re able to have this kind of discussion with some humility.