The Andon cord is literally a cord which can be pulled by every production’s worker to signal problems in production. The effect is to immediately hold production and stop to fix the issue. This concept stems originally from Lean Manufacturing or the Toyota Production System. This is similar to quality gates in software delivery pipelines. If such a pipeline fails in some way, either because a quality gate cannot be passed or because of some automatic tests are not green, or if the pipeline itself is broken (e.g., because of hardware issues) everything else will take second priority and the pipeline will have to be fixed.
I want to relate a story I read in the instructive book “working backward” by the two ex Amazon managers Colin Bryar and Bill Carr. Every Amazon employee above a certain level has to work as a customer service representative for a few days every two years. They even did not exempt Jeff Bezos from this rule. When he listened to a customer service worker on the first day of the trainng, a call came in and the customer service worker already had a hunch it was going to be a complaint about a certain lawn chair. When the case had been handled, Jeff Bezos asked how the customer service worker could guess it would be about this product and he got the answer that they had quite frequent calls about this newly listed product. His reaction was to suggest that they needed an Andon Cord for the customer service. The only option customer service had was to refund the flawed lawn chairs and Jeff Bezos took care that they got a button on their screen which immediately removed a product from Amazon’s offering once they noted there where systematic issues with it. It was much cheaper for Amazon to stop selling a flawed product and it was also better for the image of the company.
I have also often experienced similar problems with the delivery of our software projects. A unit test has failed? Let’s comment it out so we can continue shipping! We do not have the appropriate data on our test system to test the implementation for a new business requirement? Let’s skip the test and hope for the best! We need to call an external service to fully test our implementation? Well, the sandbox of this service is broken or does not have the right data or it can’t be reached because of an improper network configuration. It’s too hard to fix this so let’s promote our change to production where we can test it! All those situations call for a proper Andon Cord where you can halt your software delivery pipeline until you can do proper quality assurance again. You should never put software on the production system to be able to test it! On the contrary: Nothing, that has not been properly tested, should enter the production system. There just is no excuse for not doing proper quality assurance! Not doing it will make it much more likely that you compromise the availability or correctness of your production system. Having great developers does not automatically mean also having responsible developers! You have to explicitly work with your team to also act responsibly.
You can find more reasoning about concepts like the Andon Cord in my recently published book
“DevOps Mindset in Software Development: From Apprentice to Journeyman”
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.