Start Thinking in Terms of the System, not just Code

Often, our progress is not hindered by the time we need to implement out designs, but by wrong assumptions that make our designs flawed. It can be hard to identify such wrong assumptions because humans have a tendency to stick to wrong assumptions. The story I want to relate stems from an incident which in the end was caused by such a wrong assumption. At first glance, everything looks reasonable. But in the end, it turns out assumptions where wrong and the system wasn’t seen as a whole.

I chose this story because it is quite easy to explain, but still contains an important lesson. So lets start!

In the software system, there is a business entity Contract, which has, among other attributes, a start and an end date. Three months before the end date of a contract, a reminder should be sent to the contractor so he has a chance to renew the contract in good time.

Contracts are stored in a relational database table

CREATE TABLE contracts (

contractid CHAR(32) NOT NULL,

clientid CHAR(32) NOT NULL,

contract_start DATE NOT NULL,

contract_end DATE NOT NULL,

PRIMARY KEY(contractid)

)

Every day at midnight the contracts ending in three month are determined and reminders are sent for those contracts:

SELECT contractid, clientid FROM contracts WHERE contract_end BETWEEN NOW() + INTERVAL 29 DAY AND NOW() + INTERVAL + 30 DAY;

That should do it, right? Well, it passed all acceptance tests with flying colors and the business was happy with the handling of the reminders for quite some time.

So, everything was fine until it wasn’t anymore. Some day, the business filed an incident. They named a contract which would soon expire, but the reminder email wasn’t sent! Business wasn’t happy!

Of course, as programmers, we first tried to find a bug in the code that was executed every day at midnight. For testing purposes, we changed the contract_end date to today plus three months and executed the reminder job manually. Everything worked as it should, so there was nothing wrong with the data. The code worked and there was nothing unusual with the data so why did this peculiar email not go out when it should have? It was a conundrum! But just maybe we analyzed the problem under the wrong assumption that the regular reminder job was actually executed? Sadly, the job did not log when it started executing but only a meager ‘Done.’ when it was done. To make things even worse, there where several other scheduled jobs that were similarly tight-lipped and also jut put an entry ‘Done.’ into the same log file to obscure the fact which job actually logged this information. The decisive hint came from the time stamp. On most days, there was a ‘Done.’ shortly after midnight, but not on the day when the 30 day countdown towards the contract_end date of our peculiar contract started. So, for some reason, the job wasn’t executed and our assumption that the code gets executed on every day proved to be wrong!

We did a few things to improve the situation. Of course, more sensible logging output was added, so it was easier to see that the job actually started and how many contracts had reached the start of the three months countdown. And then we took care that a reminder also would be sent on one of the next days if the job did not execute on one day. To avoid sending multiple reminders, a field was added to the database table, which signified that the reminder had been successfully sent out:

ALTER TABLE contracts ADD COLUMN reminder_sent BOOL NOT NULL DEFAULT FALSE;

The code to select the expiring contracts was changed to take every contract into account which would expire in the next three month but did not yet have a reminder sent:

SELECT contractid, clientid FROM contracts WHERE contract_end BETWEEN NOW() + INTERVAL 0 DAY AND NOW() + INTERVAL + 30 DAY AND reminder_sent = FALSE;

When a reminder has been sent successfully, it would be updated accordingly.

The whole story reminds me of a good time I had when on-boarding a new colleague who was really curious and bright. During his on-boarding phase, we had daily one hour sessions discussing what he needed to know about our product and his tasks. Often, we reached a state in the discussion where confusion arose and most of the time, it was caused by some wrong assumption. “Let’s always check out assumptions” was the most frequent phrase I told him during that inspiring time.

Of course, the solution described is not perfect, but pragmatic. Is it good enough? Let’s give it a quick thought. What if our scheduler fails to run our reminder job for three months in a row? The reminder would not be sent, indeed. But then we have a completely different kind of a problem. Maybe our whole system has been shut down and we should take care of other problems than continuing riding dead horses and chase after problems that don’t matter anymore.

You can find more lessons of how to focus on business value and deliver what actually matters in my recently published book

“DevOps Mindset in Software Development: From Apprentice to Journeyman”

available e.g. at Amazon.