[paper review] Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection

[paper]

There are many papers, in a field called "ML4VD" (ML for vuln detection), that train and evaluate classifiers to determine whether individual functions are vulnerable. As I've written in a previous review, this is a bad way to approach the problem, as individual functions can rarely be classified as vulnerable or not without the context of the rest of the codebase. We now have a paper that demonstrates this.

In this paper, the authors select 22 ML4VD papers in Top-4 Software Engineering conferences over the last five years, and find that all of them define vulnerability detection as a binary classification problem: a bad start! The authors then look at the 3 most popular datasets used in these papers (BigVul, Devign and DiverseVul), randomly select 100 samples marked as vulnerable out of each, and look at them individually.

Before they determine if vulnerable functions can actually be determined as so on their own, they have to look at the number of vulnerable functions which are not actually vulnerable at all. First finding:

> Out of the 100 functions per dataset that were originally labeled as vulnerable, only 38%-64% (Devign: 50%, BigVul: 38%, DiverseVul: 64%) actually contain security vulnerabilities.

This was already known, and is the consequence of questionable design choices in how these benchmarks were created. They're left with 152 actually vulnerable functions. How many can be identified as so without additional context? Well, 0.

They identify 5 types of dependencies necessary to correctly determine whether a function is vulnerable: dependence on (1) external functions, (2) function arguments, (3) type definitions, (4) globals, (5) the execution environment (e.g. whether a file has already been created, etc).

What about functions marked as non vulnerable in the benchmarks? They sample 90, and are able to create a context in which these functions are vulnerable for 82 of them.

All in all, that's 8 good labels out of 390.

Then there's the question of why ML4VD papers still reported relatively high accuracies. They hypothesize it's because the models learn to pick on spurious correlations. They train a simple gradient boosting classifier that achieves 63.2% accuracy on Devign using only word counts. The paper isn't very explicit here, but heading over to the open-source implementation shows that functions are tokenized and then converted into a vector (the size of the vocabulary) with the token count for each token (so, basically a token histogram).

I'm still confused after reading this part and I think it would have deserved to be more developed: I don't really get where the spurious correlations might be coming from. I played a bit with the code and found a few interesting things:

• I could replicate the 63.2% accuracy. It corresponds to an F1 score of 52.8%.

• there are 55% negative labels in the test set, so our baseline accuracy is 55% (F1 score of 0%, though).

• there are 25 functions which are present in both the train and test set (0.5% of the test set). This should help any model a bit... Well, no, because they all have different labels in the train and test set! Who knows what other horrors can be found in the Devign dataset.

• so I think it's possible that the "spurious correlations" could actually be related to some similar dataset horror, such as some functions being basically duplicated between train and test set except for some whitespace, or something like that. The Primevul paper does have a deduplication step where they remove all whitespace before computing hashes, which hints that something like this could be going on.

Overall, a paper to be grateful for. Future work could also investigate how so many binary classification ML4VD papers could be written and then published in top software engineering conferences.