Prioritisation scores are suggestions, not rules

This week’s question was inspired by a discussion on LinkedIn. A product owner mentioned their team has a prioritised backlog of tests, uses a scoring system like PXL, yet doesn’t actually develop tests in that order. Is this normal?

Spoiler alert: we say, yes.

But if that’s the case, what’s the use of a prioritisation framework if you’re going to ignore the order it gives you? Let’s explore this a little deeper.

Neither I (it’s me, Iqbal, hello!), nor Matt, necessarily pick up tests in their strict prioritised order. Instead, we see these frameworks as a way to:

“describe experiments.” — Iqbal

and to…

“more robustly collect metadata about your tests.” — Matt

To understand why this view is so helpful, consider this scenario where you wanted to maintain the momentum of your experiment program, and you find yourself blocked on the development of some tests. Describing tests with “Ease” scores means that:

“it was really easy to look into the backlog and find an experiment that’s got a high ‘Ease’ score and kind of go: that’s a test I can develop while I’ve got some downtime.” — Iqbal

Frameworks like PIE, PXL, or ICE (that name didn’t age well, right?) are essentially a form of decision matrices. You have experiments as the rows of a table, and the criteria are the columns.

The criteria are objective ways of describing your tests, using numerical scores. The criteria should also be relevant to your team/company/product.

Ultimately, if you regularly find yourself not picking up tests in the scored order, it may be a signal to review this criterion. These frameworks shouldn’t be static.

For instance:

“someone that’s just coming in or building a new team... would probably want to audit the prioritisation.”—Matt

But it doesn’t only happen when someone is new.

“you do need to review your prioritisation matrix [there is] a change in the business goals... or even a shift in the problem-scape.”—Iqbal

The goal isn’t to create a rigid priority of tests, but rather a descriptive system that empowers decisions about what test you’re going to build, and why you’re going to build it.