DebugBetter is changing
After 100+ emails, I am pausing writing for a bit. If you have any debugging stories or questions, please send them my way. For now, I will be working on debug flashcards, an online course, and more. Thanks for reading!
DebugBetter - Level-up your debugging skills!
Recent content on DebugBetter - Level-up your debugging skills!
Latest posts
Here be dragons
There are areas of a codebase that are more…involved than others. Where there is technical debt. Where things are particularly complex, sensitive to change, or undocumented. It happens, and it’s best to give folks a warning. A previous company I worked for had a large comment block in a certain part of the codebase that started out with, “Warning: Here be dragons”.…
Debugging in 2 or 3 dimensions
When you are trying to figure out what isn’t working as expected in computer graphics, check your orientation. Left and right, up and down, these could be reversed from what you were thinking. I know it sounds simple. But you could be caught up in texture projections on rows of objects, and the whole time you were looking at the wrong object. Like I was today. When you’re just staring…
Debugging a crash leads to debugger bugs
Investigations can yield unexpected results. In this article, Bruce Dawson investigates a strange crash. He shares each step of the way, including finding unexpected issues with several debuggers. It’s a great read.
Washing machines and workarounds
Have you ever used shared laundry facilities? The machines in my building have few controls and limited instructions. You can put coins into a slot and push them in and out, and that starts the machine. You can choose the temperature. Opening and closing the lid will also pause or resume the machine. There is a light indicating if it’s running. There is a bug with these machines. Not…
Debugging needs a little extra
Beyond the compiler or toolchain support for “debug mode”, consider adding optional debug features to your systems. Here’s a trivial example: the observer pattern. You can implement a history of “previous observers” for observables. You wouldn’t ship with this enabled, but it could save a lot of time debugging instead of trying to reconstruct the history…
Bug tracking
Bugs need to be tracked somewhere. You want to know what the problem is, how to recreate it, the conditions (version, operating system, hardware), who might be working on it, and the status. It’s a similar set of information for tracking regular work tasks. I’ve used Bugzilla, DevTrack, JIRA, probably more I can’t recall, and unfortunately spreadsheets. Some companies have…
Silent failures
Bugs can occur with no errors, warnings, or feedback. When you press a button and nothing happens. No error dialog. No crash. No log written. If it’s your code, it still may not be immediately obvious what is going on. You don’t litter the “happy path” with log messages because most of the time they aren’t useful and may cost you. When you are faced with these silent…
Purposeful crash context
There are situations where you want to crash on purpose. Some code path was taken that should never be possible. Critical configuration or a key asset is missing. You want this to be known then and there: something really bad was already determined, and further execution could lead to vague failures that aren’t the root cause. Often projects will have a custom assert system in place, where…
Heisenbugs
Bugs that change or disappear when you inspect them are called Heisenbugs. It’s a pun from quantum physics where observing a state alters the state. These kinds of bugs can be confusing and disorienting at first. A bug is reported, and you can repro it locally. You go to investigate in a development or debug build, and now you can’t reproduce it. What gives? Common reasons for this…
Distance to debug
When thinking about debugging an issue, the degree of difficulty could be visualized as a distance. The longer the distance, the fewer tools you have available, and the harder (or impossible) it is to use them. The shorter the distance, the easier it is. Ability to attach a debugger (Is the application open to attachment?) Ability to change code (Can you change anything to add logging or test a…
Bugs and tech evaluations
Six years ago one of my teams was evaluating game server hosting/containerization solutions. They tried one from A Well-known Service provider. Going through a test project, they ran into multiple crashes. Unsurprisingly, they very quickly moved on to other vendors. Bugs cost sales in B2C and B2B. They erode trust. The thing is, people tend to tolerate issues in the software they develop. Local…
Bugs cost sales
Bugs will cost you sales. Last year I was trying to buy a gift from the web store of the oldest corporation in Canada. I added the item to my cart. I entered all my payment (credit card) information in correctly. I pressed purchase - and nothing happened. Knowing that sometimes there is a delay and some implementations can do bad things if you spam buttons, I waited. But still, nothing happened,…
Rob Pike on debugging
Rob Pike needs no introduction. He gave an interview in 2012 where he shared a story about debugging and what he learned from Ken Thompson: If you dive into the bug, you tend to fix the local issue in the code, but if you think about the bug first, how the bug came to be, you often find and correct a higher-level problem in the code that will improve the design and prevent further bugs.
Compiler bugs
Compilers are software, and they have bugs. This can be extra challenging as a developer because you expect your tools to work correctly, especially with something like a compiler that you know is constantly used and relied upon by many people. For example, Microsoft published a guide for how to report a bug in their C++ compiler. It contains a lot of great advice for bug reporting in general,…
Stages of correctness
Software projects are built in passes. Static analysis (lint?) time. Compile time. Link time. Test time. Run time. Correctness checking exists at each stage. Pushing it earlier leads to more explicit code, like type safety. Pushing it later incurs performance penalties. Not everything can be tested at the earliest stages. Starting with static analysis, you can verify style adherence, type safety,…
Confidence and debugging
Debugging is difficult. You have some steps that produce an error or the report of an error without steps. You’re trying to fill in the gaps. It can feel overwhelming. The investigation takes you into code and systems you’re unfamiliar with. Now you’re tracing an issue and trying to decipher and understand a new area. Taking a step back, this might lead you to look at a bug and…
Guarding from overruns
When you manage memory, mistakes can happen. Writes could happen outside of dynamic buffers or even beyond the limits of the stack or stack frame. These are very hard to track down, as they can happen with no noticeable error (write the same value that was there) or cause data corruption that crashes much further into execution, or in some completely unrelated area of the code (where it seems…
Don't give up
Debugging is hard. It’s easy to spend an endless amount of time investigating or to get completely stuck. Shrugging your shoulders or throwing your hands up will never solve the issue. It’s an opportunity to learn, and the path to the solution may not be linear. Build and use your array of techniques and skills to find a way forward. Instead of giving up, change tactics. Sometimes that…
Debugging dates and times
It’s a new year, and a good reminder of all the complexity of date and time calculations. Last month, Debug December had a challenge to determine whether a year is a leap year or not. I had to look up the logic, and it was more complicated than I expected. I also find timezone conversions and calculations around dates unintuitive. Maybe it’s because of several dimensions of non-base-10…
Debugging compilation errors
Are syntax errors, compiler errors, linker errors, or test errors bugs? They are caused by defects in code, and you can apply debugging techniques to resolve them. You might think, “just write the code properly, and this will never be an issue”. But they appear when you are porting/supporting another platform, using a different compiler, upgrading your tools and dependencies, and other…
Debug December
This month Sentry ran a series of debugging challenges from the 1st to the 24th. They were all presented in-browser and in JavaScript. There were hidden test cases that would verify your solution. You could use console.log to log messages, but there was no debugger to attach and inspect with. Some challenges used functions that were defined somewhere else, with no code or specifications. Some of…
Minidumps
When debugging crashes, you can use a crash dump to find out what happened. It’s like a video game “save game” file. A debugger can then load the dump, and it’s as if you had just encountered the crash with the debugger attached. This way you can load and inspect crashes from other people. Think about what kind of data you need to make this happen: call stack, state of…
Bugs in slow motion
“Debuggers don’t remove bugs. They only show them in slow motion.” – Unknown This is true! They are an observation and investigation tool. Features like pausing execution and stepping through code allow you to inspect snapshots in time. Despite being called a debugger, they only help you diagnose them. They are still a critical tool; sometimes diagnosis is the hardest part!
Debugging Super Smash Bros.
Masahiro Sakurai was the director of the Super Smash Bros. games and several Kirby games. He recently started a YouTube channel sharing all about game development. In one of his videos, he reviews the “Debug Mode” features that were created for Super Smash Bros. In these games, the debug features could be used on-console (devkit) with a second controller. He recommends supporting them…
When the commit message is wrong
You are going to read a commit message, then look at the code diffs, and realize they don’t match. Often it’s something innocent, like simply being incomplete. There are more changes and fixes included than are mentioned in the commit message. Other times there are changes in formatting or something else that result in no compiled or runtime change, yet the commit message says it fixes…
When the why behind code is missing
“Misc optimizations” “Updates” “Fixes” When documentation is missing, you only have the code to go by. Even if the original author is available, they may not remember why they wrote something the way they did. Even if you are the original author. It makes debugging harder than it could be. Do better by documenting whatever you discover, even for code you…
The why behind code
You’re debugging something and come across unfamiliar code. It’s doing some things that don’t make sense and could potentially be the source of the bug, but you aren’t sure. This happens a lot with old codebases. Why was it written this way? It isn’t always apparent. Using version control, you can look up when the file or even lines within it were last changed. When…
Performance issues and bugs
Where is the line between a performance issue and a bug? Does a line even exist? When something is slow, you can add a loading bar or animation. If it’s slow enough, it’s perceived as stuck - that’s a blocking bug. How about when you’re watching a video, and it’s constantly pausing because the hardware can’t keep up? That seems like a bug - the user experience…
Dogfooding
Dogfooding is when a company uses its own products or services. I remember this term originating from Microsoft, although Wikipedia has some evidence linking it to actual dog food. This has many benefits, including finding a variety of bugs. It isn’t a complete solution, as it doesn’t address extreme edge cases. It also doesn’t match the scale of coverage that would come from a…
Debugging out of order bugs
Some bugs are the result of operations happening in the wrong order. These usually show up in distributed, asynchronous, or parallel systems. They can also be intermittent, making them extra difficult to debug. The first thing you want to do is understand the complete order of execution, and logging is a good place to start. If you can’t log centrally, make sure log messages have a timestamp…
Sometimes you have to break it
When something isn’t working, your changes don’t seem to have any effect, and you can’t use logging… break something. I don’t mean grabbing your monitor and throwing it. Some environments are difficult or impossible to log messages from. Graphics shaders, managed cloud services (parts of them), nested tooling for building web bundles (scripts calling CLI calling…
The tale of the invisible gryph-chargers
During the development of Warhammer Age of Sigmar: Storm Ground, there was a bug where part of a unit was invisible. Specifically the Vanguard Palladors, who ride “Gryph-chargers” like horses. So these soldiers were floating around. The game would run and function correctly, but their “mount” (the Gryph-chargers) was not visible at all. This only happened in the client…
The Best Code is No Code At All
In an age where generating more code, and more code faster, seems to be a goal: “The Best Code is No Code At All” This is a quote from Jeff Atwood. He wrote this article about the topic in 2007, and the next year started Stack Overflow. Yes, the Stack Overflow. Code has the potential for bugs. More of it is more risk. It’s also more to maintain. This has been a challenge in…
Off-by-one errors
An off by one error could mean the difference between zero and one. Or zero and 4294967295. Or a and b. True or False. They are often subtle to human perception but can have drastic implications for code execution. They appear often when dealing with ranges and indexing. Does the programming language index from 0 or 1? Did you mix up > with >=? Ceil and floor? They result in the wrong object being…
The language of bugs
Written or verbal bug reports use words and phrases like: Crash Halt Freeze Jank Stutter Not working Blue screen Error Stuck Access Violation Shut down Stopped Broke Glitch Blow up Illegal instruction Restart Some of these are more useful than others. The follow-up questions are when, where, how, etc. This is similar to the vocabulary of emotions. A wider vocabulary allows for more accuracy and…
Record bugs somewhere
“Hey, the button on the second page doesn’t work.” “Oh, by the way, when I put numbers into this field and press submit, nothing happens.” You get busy, and a few days later - what was the bug again? If it was mentioned in a meeting, I know I’ll forget the details unless I take notes. Sure, you can review Slack messages and emails, but the bug reports are…
The tale of the one-word commit message
One morning a programmer came into the office. They got a cup of coffee, sat down and updated their local repository. Their email notified about a crash bug had just been assigned to them. It was logged a few months prior, and only recently triaged. They read the reproduction steps and reviewed the attached crash dump. A few minutes later they verified the steps work, and the dump matched their…
Given enough eyeballs, all bugs are shallow
The longer form is: “Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.” This is Linus’s law, and was named in honour of Linus Torvalds. I’ve seen examples of this at work, but also tricky bugs that took entire teams weeks. What has been your experience?
Bugs that hide other bugs
When you have bugs that block access to sections of your software, other bugs start to accumulate there. This often happens with projects in development where it’s more acceptable to say, “This is broken right now”. The trouble is that while that area is inaccessible, nobody can fix the known bugs in it. New ones will crop up from things like external API changes or refactoring…
Incompatibility is not a bug
Have you ever tried to get something to work, only to encounter some strange behaviour, cryptic error, or complete failure? Then after investigating, you find out it’s not compatible. You were trying to run with the wrong version of something else. It half-worked but actually will never completely work. This isn’t fun or a good use of time. It’s debugging, except there…
When to crash
The most severe log level is usually something like “critical” or “fatal”. This means execution cannot continue. When should you use this? Often it’s an unrecoverable error, like running out of available memory. It gets tricky when it’s something like crucial data, like key configuration data. Can you fall back to defaults? Can the application still function in…
Log levels
Not every log message is useful. Log levels help you categorize messages, so you can filter out just what you need. You want logging to be useful, and too much isn’t always better. Use the granularity that best matches your needs. Follow the best practices of tools you build that have log level support. It’s no fun when you start monitoring for errors and every existing log message is…
Debugging and cleverness
Brian Kernighan is a computer scientist who famously co-authored the first book on the C programming language. He is also the originator of “Hello, world”. One of his quotes about debugging is: “Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it?” In…
Death marches
Sometimes a project has a mountain of bugs to fix, and it takes months to work through them. I’ve heard this described as a “death march” (an unfortunate use of the term), because it’s generally unpleasant work often accompanied by overtime, stress, and pressure. There are lots of reasons why this happens and lots of advice about how to avoid it. I don’t have a magic…
60% of the time, it works every time
Bugs that occur only some of the time add another layer of complexity (and frustration) when debugging them. One cause of these is timing issues. Asynchronous systems are particularly susceptible. Any sort of magic timeout will be problematic. Best practices like callbacks are key. These issues show up in application code and in tests. A flaky test, which is one that sometimes passes and sometimes…
Reduce permutations, reduce bugs
It’s tempting to add another boolean as a function parameter. Or as a property/member/state variable. It’s just a simple flag, right? bool isSuper; bool isDemo; bool canAccessFeatureAwesome; bool isStudent; bool debugMode; These five variables produce 2^5 (32) possibilities - possibilities you need to test and support. With this increase in possibilities comes an increase in risk of…
Key bugs
It’s common to use strings as keys for map or table data structures. It’s also common for humans to make typos or find it very hard to see the difference between "Boss_0_Attack" and "Boss_0_Atack" or "Boss_0_attack" or "Boss_O_Attack" 100% of the time. With some map implementations, you won’t get an error about an unknown key, but a new one will be silently created instead.…
Bugs and Tuxedos
David Henke was the SVP of Engineering/Operations at LinkedIn and Yahoo. He joined LinkedIn in 2009, when it was experiencing massive growth. He has a number of quotes about software development, including: “A feature is a bug in a tuxedo.” I think this is similar to “The only difference between a flower and a weed is judgement.”. I’m still thinking about it, though.…
Debugging in Godot
Godot is an open-source game engine. You can write code using C# and C++, but most often folks use GDScript. GDScript is a Python-like language created for Godot. It’s quite easy to use and is integrated nicely with the editor. And there is a debugger for it! By default, the debugger is attached when launching your game within the Godot editor. You can create breakpoints in your .gd files,…