FuzzBench The fuzzer benchmarking platform

Journal Special Issue on Fuzzing:
What about Preregistration?

22 Apr 2021

co-authored by Marcel Böhme (Monash University), ‪László Szekeres (Google), Baishakhi Ray (Columbia University), Cristian Cadar (Imperial College London)

We think that the incentive structure for fuzzing research is broken; so we would like to introduce preregistration to fix this.

Preregistration is a publication model whereby a submitted article is primarily evaluated based on (i) the significance and novelty of the hypotheses or techniques, and (ii) the soundness and reproducibility of the methodology specified to validate the claims or hypotheses. The actual evaluation or experimentation (apart from some supporting preliminary results) is conducted only after the paper has been in-principle accepted. The final acceptance will depend only on the methodology that was ultimately followed, not the final results.

Why Preregistration?

These days, to get your fuzzing paper published, you will need to submit strong positive results. We believe this unhealthy focus is a substantial inhibitor of scientific progress:

  1. Duplicated Efforts. A lot of important investigations are never published, which leads to duplicated efforts across groups. Even though a hypothesis or approach is reasonable and scientifically appealing, if the hypothesis proves to be invalid or the approach turns out ineffective, an interested research group will only find out after reproducing these experiments.
  2. Overclaims. There is an incentive to overclaim the benefits of an approach, which makes it difficult to reproduce the results and misinforms future investigations by the community. Authors are uncomfortable to share their research prototypes, which impedes reproducibility and future work. For instance, in 2020 only 35 of 60 fuzzing papers that we surveyed were published together with the code.

Furthermore, sound fuzzer evaluation imposes a high barrier to entry for newcomers. A sound fuzzer evaluation requires (i) a well-designed experiment methodology and (ii) substantial computational resources. Because fuzzing is inherently random, there can be a huge variance between the outcomes of a fuzzing campaign. To control for this randomness, the experiment must be repeated a sufficient number of times. To validate a claim, a fair baseline must be chosen. To establish superiority over the baseline, pertinent measures of effect size and statistical significance must be computed. To increase generality, fuzzers must be run on a representative set of benchmarks with a reasonable time budget. The repeated evaluation during fuzzer development may require CPU centuries. Researchers may not have the required computational resources. Newcomers may not be aware of the pitfalls of experiment design and find out only after receiving the reviews and after conducting costly experiments following a flawed methodology.

By switching to a two-stage publication process that separates (1) the preregistration and (2) the large scale experiments, we can remedy both issues:

  1. If the Registered Report is in-principle accepted and the proposed experiment methodology is followed without unexplained deviations, the experimental results will be accepted as they are. This minimizes the incentive to overclaim and allows publication of interesting ideas and investigations irrespective of experimental results, thus including useful negative results.
  2. The authors get early feedback on the significance and novelty of the main hypothesis or approach as well as the soundness and reproducibility of the experimental methodology. To further lower the barrier to entry, we will also provide additional support with fuzzer benchmarking as detailed below.

We are hoping that our initiative will turn the focus of the peer-reviewing process back to the innovation and key claims in a paper, while leaving the burden of evidence until after the in-principle acceptance. The reviewers’ focus shifts from gatekeeping to productive feedback, and is aimed at ensuring the best study design possible.

Fairness. Evaluating the study design rather than the final article precludes some types of subconscious bias, such as publication bias (where authors’ are inclined to selectively publish only positive results), confirmation bias (where reviewers might give more credence to results that support their own views), and impact bias (where reviewers might give novel results more consideration).

Support. In this first-of-its-kind joint venture between academia and industry, we are pledging our support with the required infrastructure and resources, if needed. This is provided by Google’s fuzzer benchmarking platform and service, FuzzBench. Our aims are to reduce the barrier to entry, to facilitate a sound experimental evaluation, to maximize reproducibility, and to facilitate innovations towards a more secure open-source software ecosystem.

Reproducibility. By requesting prospective authors to specify a precise experiment protocol and consulting an artifact evaluation committee, reproducibility becomes a first-class citizen of the reviewing process. We also request authors to provide the final artifact, to ensure that other researchers can build on these results. Current authors would receive early feedback on the soundness and reproducibility of the proposed evaluation methodology while future authors would have clear instructions on how to reproduce the results.

Special Issue on Fuzzing

As a first step, we are planning to organize a Journal Special Issue on Fuzzing. Our main aim is to remove the current barriers, and make it as easy as possible to publish ideas with robust, bias free, and easy to reproduce experiments and evaluation results. To this end, we propose a new publication model that is based on preregistration, i.e., the practice of registering a scientific study before it is conducted [more details], and that is backed up by infrastructure and resources for carrying out large scale experiments, if needed.

Stage 1 (Preregistration)

  1. Stage 1 submission. The authors would start by submitting a registered report, a full technical paper sans the experiment results. However, to assess the feasibility of the experiments, we expect some preliminary results in small-scale experiments. Apart from fuzzing techniques, we very much welcome empirical or reproducibility studies, novel measures of fuzzer performance, well-grounded fuzzer evaluation guidelines, and related areas such as automated debugging and repair of the discovered software bugs.
  2. Stage 1 reviews. A registered report is assessed by at least three peer-reviewers in terms of novelty, significance, soundness, reproducibility, and presentation. In the absence of results, the soundness of the experimental design, including a discussion of potential threats to validity, is critically assessed.
  3. Stage 1 notification. The authors will be informed about one of the following outcomes: in-principle acceptance, rejection, or (minor/major) revision. A report could be rejected, e.g., for lack of novelty, low significance, or poor experiment design, but not for the results of their experiments.
  4. Optional public availability. If the submitted report is in-principle accepted, the authors can opt in for us to make publicly available the registered report.

Stage 2 (Experiments and Final Review)

  1. Executing the Experiment Protocol. After the in-principle acceptance, the authors have sufficient time (4 to 6 months) to execute the specified experiment protocol (with an allowance for a reasonable departure from the specified protocol). If needed, authors are provided with the required resources to carry out their experiments via the FuzzBench platform. FuzzBench provides both the infrastructure and computational resources to evaluate fuzzers. Researchers can submit their own fuzzers, benchmarks, or modifications to the platform itself. It’s all open source. Stage 2 submission. The authors submit an artifact and update the Stage 1 submission to accommodate the experimental results, their interpretation towards the main hypothesis and key claims, and their adjustments to the experiment protocol that was actually used. Of course, authors are welcome to further improve other aspects of their submission and artifacts. The authors can ask for one extension. If the deadline elapses without a Stage 2 submission and no reasonable request for extension was submitted, the preregistered report is rejected.
  2. Stage 2 artifact submission. Together with the paper, the authors submit an artifact, which contains (a) the data and results that support the claims in the paper, and (b) the tools and datasets required to reproduce these results.
  3. Stage 2 artifact evaluation. This will be conducted by a dedicated Artifact Evaluation Committee (AEC), similar to those used by many top conferences.
  4. Stage 2 reviews. This is expected to be a lighter review stage, where reviewers check that the agreed experiment protocol has been followed and results correctly analyzed. The reviewers can engage the artifact evaluation committee to check to which extent the claims in the paper submission are supported by the experiment results.
  5. Stage 2 notification. The authors will be informed about one of the following outcomes: in-principle acceptance, (minor/major) revision, or–in very rare cases where the proposed experiment protocol has not been followed–rejection. The outcome does explicitly not depend on whether the experiment results are positive or negative.
  6. Camera-ready paper and artifacts are published.

Survey (What do you think?)

We would love to hear from you about this publication model. Please consider filling out the short survey.