Creating a Safe Culture of Rollback
Devops is a hot topic in the industry currently and one I have taken more interest in lately. Having returned from attending the SLC Devops Days this week I have been reinvigorated to dig deep and push for an improved culture of Devops at my place of work. The overarching theme, and one that is not new by any stretch of the imagination, is that devops is a culture. It has been talked about many times before but I had never really taken to heart that the culture is where its at with devops. The tools will follow, but if you only have tools but not the culture you are no better off. It is very interesting, as I have read stories of companies that have successfully adopted a culture of devops and as I have started to read books on devops like The Devops Handbook they talk about how there is an improved level of happiness across the organization when you have successfully adopted the culture. Less heroics, less fire fighting, less stress, etc. At the same time it is such a paradigm shift that it can be hard for people to accept. People get used to the way that they work, even though they know it doesn't work people hate change. This is crazy to think about, people are resisting the change that will make their life better. A lot of times they even know that this is what they are doing?
As my company has moved to a more mature devops culture it has been fun to see the changes to the technology of the delivery pipeline. However more interesting that that is the change to the culture of the organization and how people has accepted it. In the last few years we have gone from deploying once every few years to every three months, to yearly, back to every three months. There have been some awesome improvements over the last year. However there is still a long way to go.
As we have geared up for an upcoming release recently we ran into an interesting problem. We have a continuous delivery pipeline that is suppose to serve as the gatekeeper to our mainline of code and it has done a fairly good job. Well a perfect storm happened where we had two changes go in right next to each other that ended up causing us some trouble. The first one went in caused some issues with our database deployment time. This particular change made a 30 minute process take 10 hours. Not a great option. Well the fixt was already in progress before the problem was discovered so we failed forward, put the fix in and were ready to branch the code. Well we had a similar but different problem start happening. Did the fix not work completely? Were their environment issues? No one really knew. We finally ended up basically running a git bisect and finding a commit that had the issue. I notified the committer and he agreed that we needed to pull the change as soon as possible. This was the first experience with pulling out a coworkers code. He was sad that the change would need to be pulled, frustrated with the issue, but passionately agreed with the revert. We also had another change that was a pretty big change to one of our business processes that could allow a user to get into a bad situation. It was a good change but a change that had hard to find, far reaching effects. Well the decision was made that we should pull this change out as well and work on fixing the far-reaching effects. Well this change didn't go so well. The developer said it would be too hard to pull out and that we should just fail forward and by the time that I became involved we were two days into the failing forward. I ran through and figured out it would take maybe half an hour to revert. This is what I did and the release moved forward. However pulling out this change had a drastic effect on the developer that put in the recent change and as, more or less, a tech lead on his team got the team to just give up on the feature and not put it back in.
What an amazing difference between these two people. One understood that we need to rollback quickly in the event of a failure. The other was stuck in their old ways of thinking in that they didn't want to rollback. Just goes to show that even though a devops, experimenting culture will be best for everyone, not everyone sees this. There are of course other issues highlighted in this story. We should have a more robust pipeline that shouldn't allow these things in. Better telemetry in being able to pinpoint issues. All of these things need to be worked on however, as was said before, we need to fix culture first, or at least in parallel with our technology changes.