From blame to learning: one postmortem that stuck with me

In my blog post "The first time I broke production (and what happened after)” I described an experience with wrecking the productive system of a web shop with quite a bit of revenue. We did not know about blameless postmortems back then. It would have been a prime example for doing such an exercise. So, what is a blameless postmortem? It is a structured way of documenting incidents and their resolution once the immediate stress of an ongoing incident is gone. You want to write down what happened and how the issue got resolved to preserve the knowledge for your whole organization. It is important to do no finger pointing to help people share everything they know and not hide their knowledge out of fear to get punished. I discuss this tool in the book “DevOps Mindset in Software Development” as an important exercise in reacting to issues.

Now is the time to show an example of a real postmortem. Humans learn best by example. A more extensive postmortem would also contain a timeline of the events that took place but nevertheless, the postmortem below shows very important aspects of this practice:

1. It does not try to push the blame on some person

2. It describes the nature and the impact of the incident

3. It describes how the root cause was identified

4. It includes action items how to avoid such incidents in the future

Here is the sample postmortem. Enjoy!

Date: 17.08.2021

Author: @Nic

Impact: Critical - Production deployment failed due to DB migration issue and shop was down for 3 hours.

Root causes: DB migration was trying to add 3 new columns into orderitems table, which contained more than 100 million records. The ALTER TABLE statement which was used to extend the table with three new fields tried to copy data into a temporary table which took way too long (more than eight hours)

Triggers: During deployment, we were gettinga a timeout error because the DB migration was taking too long.

Resolution: We analysed the order data present in the system and we noticed that it contained all order date from the the date the shop went live onwards. Together with the customer, we decided to remove old orders from the system. This was sufficient so that ALTER TABLE statement could be executed in an amount of time short enough so that the deployment would not stop because of a timeout. Detection: Manual

Check table size before changing the table schema.

If possible then keep cleaning unnecessary data from the database to improve the performance and avoid any hassle.

If possible try to keep staging system DB similar to production, so that we can detect such issue earlier and act on those.

What went well:

The output of the deployment job output helped us a lot to understand and dig deep into the root cause of the issue.

What went wrong:

We had tested the code on staging system and never noticed production system database size. We should keep in mind production database and related data while writing database migration script.

Where we got lucky:

We were lucky to roll back our changes and get up production shop live quickly and without any impact.

Solution:

We have developed a small script to cleanup data from oxorder and oxorderarticles tables. We have also decided to keep cleaning up order data older than 3-4 years. We are also planning to scan and check other tables to cleanup unnecessary data, so that the overall performance of the production shop does not degrade unneccessarily.