Stop Getting Surprised in Production: Here's How to Nail It Every Time

Today, we're zeroing in on a critical stage in the product development cycle - acceptance tests.

This is perhaps your most systemic and scalable way to eliminate surprises in production.

I have not worked with a company that is strict about acceptance tests. In fact, in Israel more often than not, companies have no acceptance at all or very vague sanity tests.

This is no bueno, and I can guarantee it’d come back and bit you in the…

Not to confuse with technical tests or regression tests, acceptance tests are the PM’s last line of defense and are ran by the product team.

You, the PM, will be in the room as well as the squad’s designer. At the very least.

These are the hard-earned lessons I’ve learned over the years.

• Share in advance your test’s checklist with the team.
  This will save you a lot of time, as the expectation are set and this allows the team to make sure that what’s important to you works. This will be a shortlist of what you’d ask them to show you (device types, specific interactions.. etc..).
  

• Have your squad’s designer testing features on the intended device.

  • Don’t test mobile on mobile-web, which is a common mistake.
    Test mobile products on mobile devices only. If you don’t, you will run into unexpected behaviours in prod. I promise.

  • Have your designer to approve UI and interaction aspects.
    If you are working with a strong designer, they will notice discrepancies that you won’t. More than once. They have developed an eye for those details over years - don’t miss an opportunity to use that talent.
    

• Have the engineer who worked on the feature in the room.
  Don’t test with your QA only. We want good fast and direct feedback cycles.
  

• Define Clear Criteria: there are two important to dos for you here.

  • Have a clear final design in Figma. Not a Final_last file, but really a ready for dev file or tab that is the ground truth you all agree on as the final design for acceptance. It would have to be crystal clear where is that tab to everyone and the design would have to be fully final, i.e. final microcopy included.

  • 2nd ground truth - user stories. Simply put, your stories+Figma would be the only answer for questions you run into during the acceptance. This includes behaviours, flow, UI design, interaction.
    

• Product analytics: have your squad’s analyst to make sure the events defined are firing as expected. This can be done async.

This is my take, and most companies don’t work this way. But, this is a hard-earned experience so if you use this, you might save yourself a lot of time and trouble.

Have clear close-list of possible outcomes, agreed on everyone.

The list could be something along these lines:

1. Another acceptance event should be scheduled - this is the least favourable outcome and usually means that the acceptance was too early anyhow. Feature is not ready.

2. Async acceptance should be done as a follow up - the feature is almost there, but you do have comments you want to see solved before rolling to prod.

3. Changes should be made and then deployed - relates to minor changes that you can trust your team to fix and you don’t need to check once again - wrong color, typo, indentation.

4. Approved for deployment without any further changes - the holy grail. yay!

Flowchart for Decision Making and test outcomes.

Let me know if this was useful by replying!