Towards Resource-Aware Security Testing of Software

As software permeates every facet of life, it is imperative to assure the safety of software systems. Software vulnerabilities—exploitable software bugs—allow an attacker
to destroy privacy, steal identities, and even extort money from victims. Therefore, software bugs must be discovered before an attacker can exploit them. This dissertation presents our work on mutational fuzzing, a software testing technique for finding software bugs. Specifically, we argue that the efficiency of mutational fuzzing can drastically change depending on its parameters, and thus, automatic parameter optimization can help in improving the fuzzing efficiency. We validate this argument by designing, implementing, and evaluating several systems that employ novel techniques optimizing parameter selection for mutational fuzzing. Our specific contributions
are that (1) we precisely define fuzzing and its parameter space; (2) we analytically study the effectiveness of mutational fuzzing in terms of bug finding probability;
(3) we then address three strategies in optimizing mutational fuzzing over the parameter space in terms of the number of bugs found; and (4) we finally show a post-fuzzing
strategy that enables prioritizing security-relevant bugs under limited resources.