Improving fuzzing assessment methods through the analysis of metrics and experimental conditions

Fuzzing is nowadays one of the most widely used bug hunting techniques. By automatically generating malformed inputs, fuzzing aims to trigger unwanted behavior on its target. While fuzzing research has matured considerably in the last years, the evaluation and comparison of different fuzzing proposals remain challenging, as no standard set of metrics, data, or experimental conditions exist to allow such observation.

This paper aims to fill that gap by proposing a standard set of features to allow such comparison. For that end, it first review the existing evaluation methods in the literature and discusses all existing metrics by evaluating seven fuzzers under identical experimental conditions. After examining the obtained results, it recommends a set of practices –particularly on the metrics to be used–, to allow proper comparison between different fuzzing proposals.

Keywords: #Fuzzing, #Evaluation methodology, #Security, #Software testing, #Metrics

Authors: Maialen Eceiza Olaizola, Jose Luis Flores, Mikel Iturbe