Detecting Defects in Software Systems

Throughout my time working with complex distributed systems, I have learned that consistently detecting systemic issues is surprisingly difficult and a bit of an art. The canonical way to do this is to set up alerts that notify an engineer when something is wrong. I have personally written many terrible alerts and regret having subjected myself and colleagues to the pain of such useless alerts.…