Recent content on patch
This post describes a proposal and is a call for clients, if you will. I have an idea for an optimization in the GCC MC/DC instrumentation ( -fcondition-coverage ) that I estimate would reduce the overhead in both compile time, object size, and run time, roughly 2-3 times, and I am looking for clients to fund this work. GCC has supported MC/DC since version 14 (released May 2024) and works quite…
Graph improvements I wanted zcov to quickly and naturally show the coverage status of a piece of code. Most of the time we’re interested in missing coverage as that is where we need to do work, either adding a test, improving the spec, or figuring out if there is a mismatch between the code as-is and then goals we’re trying to achieve. zcov can now colour code the blocks to show this,…
GCC 16 will probably release in a couple of months, and comes with a couple of my patches. There’s nothing too big this time, but a couple of bug fixes and some quality-of-life changes. You no longer need to explicitly pass -ftest-coverage for -fcondition-coverage and -fpath-coverage to be useful, it is now implied. The -ftest-coverage flag controls if GCC creates the .gcno files gcov needs…
2025 in a minute 2025 was an eventful year. My prime path coverage was included and released with GCC, along with a few minor patches. I presented prime path coverage and my work on GCC at NASA at the Johnson Space Center in Houston, and published the pre-prints of my papers on MC/DC and prime path coverage in GCC. I wrote and did a course on package management with Guix. I have delivered on…
There are many good reasons to maintain a proper git history, with deliberately crafted and focused commits, a state that builds, and good messages that provide context, motivation, and reasoning for the change. A neat side effect of a well kept log is that it boosts the usefulness of git bisect . Briefly, git bisect binary searches the history to find the commit which introduced the bug, and…
One of the aspects of Guix that really sets it apart from other package managers is how there is not much difference in power between the user and the repository. Sure, you can build your own .debs just fine and manage them with dpkg, but it takes a lot more infrastructure to have your own small packages play well with apt (and other users). For sharing packages in Guix, most of the time it is…
Drawing insights We can draw interesting insights from measuring code coverage. This is a binary search that I borrowed from the internet and used in my talk on prime path coverage in GCC at TechTown 2024, very slightly revised. It returns the index of the needle (x) in a sorted array, or not-found (-1) if it could not be found. int bs ( const int A [], int N , int x ) { int min = 0 ; int max = N…
The last post used custom replacement functions and allocator interfaces to simulate allocation failures for testing purposes. Those approaches work well when you control both sides of the function call, but it won’t work if we want to test handling errors originating in libraries we don’t control. The malloc might fail because the system is out of memory or when the resource limit is…
This is a trivial C++ function that does an unchecked access of an array element. It is perfectly fine to use if i is in bounds, and the CFG is just the straight line you would expect. Testing wise, the only thing we really can do is call it with a valid vec and i in bounds, as anything else would be undefined. int & unsafe_at ( int * vec , std :: size_t i ) { return vec [ i ]; } unsafe_at() Here…
In this post we will work with the basic structure of a switch in a loop, and look at how slightly different decisions and designs affect the complexity and the testing requirements. This construct shows up in many programs, such as state machines, file format parsing parsing, command line argument processing, and event loops. It a consideration became sensitive to when I started measuring prime…
In this post we will use path coverage as a guide for testing a C function and explore how a light rewrite improves our ability to test it. Most testing is really about observing how the program responds to data, and coverage is mapping that response to the program structure. By changing the program we change the structure the data operates on, even when two programs are functionally equivalent.…
I have been wanting to type up some notes on effective code coverage for a while. I am doing a talk at NDC Techtown 2025 on a few code coverage topics, which is an excellent opportunity to do so. These notes, and the talk, will touch on a few different topics; how coverage can guide testing, on effective metrics, coverage as a concept, and how different programming styles affect the possibility…
I saw something puzzling today while using zcov – a function I didn’t write that had no coverage. In my parsing code I have a small struct, something like this: struct block { block (); int id ; int call_site ; int nonlocal_return ; std :: vector < int > preds ; std :: vector < int > succs ; }; The block constructor shows up twice in the report, and only one is covered. Where does this…
This post is based on an interesting case NASA brought me; a small test program with a phantom condition. In the example, timed_out() is always true, which means checkout() cannot satisfy MC/DC, but let’s pretend it does something interesting. The function has one decision with two conditions, and four outcomes. Let’s measure MC/DC and confirm: bool reset ; int count ; bool timed_out…
GCC 15.1 was released today , and it comes with prime path coverage support! Try it out and build your programs with -fpath-coverage , it truly is a brilliant metric. I wrote another paper detailing the implementation of the prime path coverage support. You can get a copy here or from arxiv . Abstract: We describe the implementation of the prime path coverage support introduced the GNU Compiler…
I am happy to announce that the prime path coverage support has been merged into GCC mainline . This was a continuation of my MC/DC project with NASA. I did most of the programming in the spring through autumn of 2024, followed by a long period of mostly testing and paper writing. NASA have been using GCC builds with this feature for about six months now, and they are quite happy with what they…
I gave a talk today at the Johnson Space Center in Houston, on the GCC coverage features I have done with them. The talk covered what prime paths and the motivation for prime path coverage, why they are an important development in dynamic analysis and how leveraging it will improve the quality of the testing we do of our systems. This marks the end of my two projects with them, MC/DC and prime…
My paper on Modified Condition/Decision Coverage in GCC is published (in preprint)! You can get a copy from here or from arxiv . It covers the reasning, inner workings, and algorithms behind the -fcondition-coverage flag in GCC. Abstract: We describe the implementation of the masking Modified Condition/Decision Coverage (MC/DC) support in GCC 14, a powerful structural coverage metric with wide…
Line coverage is typically the first structured metric looked at, and it provides a rough and imprecise measure of how thoroughly the test harness exercises the code. Briefly, line coverage is the answer to the question was code at this line ever run? The most important insight is finding the code that is not at all exercised in testing, but it can also be used as a coarse profiling tool by…
I am excited to announce my latest project, under the name zcov . It is a program for interpreting and appreciating coverage, with the goal of being the only tool you need for working with compiler instrumented coverage. It is still under development, but I have already been able to use it to understand and explain subtle program interactions, the path coverage view being particularly powerful.…
NASA reported an odd crash in gcov, and I found it interesting and deep enough to warrant describing the root cause and the path it took me on. The actual crash itself was quickly found with gdb; reading the block.locations.lines when printing the source lines of a path when the array was empty (with size()-1, even!). A fix would be to not do that e.g. by guarding the access, but the question is…
This year’s Techtown conference in Kongsberg is through, and I wanted to do a trip report of sorts. There were ~400 attendees, the highest attendance yet, and an all-around fantastic atmosphere. I learnt a lot and had the privilege of striking conversations with many brilliant people, on diverse topics such as language design, compiler development, compiler features, embedded constraints,…
The prime path coverage support is shaping up nicely, so I wanted to write a bit about the last series of improvements to the upcoming prime path coverage feature in gcc. gcov reports Path coverage differs from statement, branch, and condition coverage in that coverage ties to the function and not individual lines. This is quite different from how gcov has reported up until now. Furthermore, it is…
It’s time for another progress report. The last post covered the phases at a high level – please read that first for an overview. Observing paths To observe paths we associate a bitset with each function, where each bit corresponds to a path. Which bit maps to which path is simply its index in the lexicographically sorted set of paths. When an edge is taken, the a mask is applied to…
After the success of getting MC/DC support into gcc, NASA decided to keep funding the project and focus on (prime) path coverage. I have been working on this problem for a while now and it is about time to write a post on the progress so far. Prime paths The number of paths in a program grows very fast. Somewhat simplified, a path is a sequence of blocks evaluated for a run. Any control flow…
I have to admit, it feels quite nice to finally write this post. My support for MC/DC in gcc was merged April 4th, followed by a couple of bug fixes. This post is just an announcement - I plan to some time later write a more detailed post on how it works and how to use it, and maybe one on MC/DC itself. It has been a long project. I started working on it around the start of 2022, so it took a bit…
I ran into an interesting problem when adding support for Modified Condition/Decision Coverage (MC/DC) in gcc (a topic for a later post): how do you large scale test a new compiler feature? In this case, I wanted to figure out what kind of control flow graphs gcc could create, how particular graph shapes would interact with my changes, and if my assumptions actually held. Hand rolled examples took…