Fuzzing for CPS Mutation Testing

<p dir="ltr">This is the replication package for the paper `Fuzzing for CPS Mutation Testing`, which is accepted by ASE 2023.<br></p><h2><b>Abstract</b></h2><p dir="ltr">Mutation testing can help reduce the risks of releasing faulty software. For such reason, it is a desired practice for the development of embedded software running in safety-critical cyber-physical systems (CPS). Unfortunately, state-of-the-art test data generation techniques for mutation testing of C and C++ software, two typical languages for CPS software, rely on symbolic execution, whose limitations often prevent its application (e.g., it cannot test black-box components).</p><p dir="ltr">We propose a mutation testing approach that leverages fuzz testing, which has proved effective with C and C++ software. Fuzz testing automatically generates diverse test inputs that exercise program branches in a varied number of ways and, therefore, exercise statements in different program states, thus maximizing the likelihood of killing mutants, our objective.</p><p dir="ltr">We performed an empirical assessment of our approach with software components used in satellite systems currently in orbit. Our empirical evaluation shows that mutation testing based on fuzz testing kills a significantly higher proportion of live mutants than symbolic execution (i.e., up to an additional 47 percentage points). Further, when symbolic execution cannot be applied, fuzz testing provides significant benefits (i.e., up to 41% mutants killed). Our study is the first one comparing fuzz testing and symbolic execution for mutation testing; our results provide guidance towards the development of fuzz testing tools dedicated to mutation testing.</p><h2><b>Package description</b></h2><p dir="ltr">This replication package contains all the source code for <i>MOTIF</i> and <i>SEMuP</i> and singularity images containing dependencies that we used for our experiments. The MOTIF singularity image does not include AFL++ so that we can replace it without rebuilding the image. Additionally, we provide case study packages for each subject where it is open-source. The following list is the description of each file in this package:</p><ul><li><i>MOTIF.tar</i> : all the source codes for the <i>MOTIF</i> pipeline</li><li><i>motif_default.sif</i> : singularity image that we used for our experiments</li><li><i>AFL++4.05a.tar</i> : the version of AFL we used for our experiments</li><li><i>MLFS.tar</i> : case study package of MLFS for <i>MOTIF</i></li><li><i>ASN1.tar </i>: case study package of ASN1 for <i>MOTIF</i></li><li><i>SEMUP.tar </i>: all the source codes for the <i>SEMuP</i> pipeline</li><li><i>semup_full.sif</i> : singularity image that we used for our experiments</li><li><i>ASN1_SEMuP.tar</i> : case study package of ASN1 for <i>SEMuP</i></li><li><i>results.tar</i> : experiment results that are used in the paper</li></ul><p dir="ltr">Note that there are additional packages: </p><ul><li>ASN1_20231024.tar: case study package of ASN1 for the updated version of MOTIF (see <a href="https://github.com/SNTSVV/MOTIF" rel="noreferrer" target="_blank">git repository</a>)</li><li>MLFS_20231024.tar: case study package of MLFS for the updated version of MOTIF (see <a href="https://github.com/SNTSVV/MOTIF" rel="noreferrer" target="_blank">git repository</a>)</li><li>AFL++-4.09a-Ubuntu22.04.tar: compiled AFL++ for Ubuntu 22.04</li><li>motif_default_22.04.sif: singularity image for execution of MOTIF on Ubuntu 22.04</li></ul><h2><b>Pre-requisition</b></h2><p dir="ltr">We use Singularity to provide the same environment for all the users. Users who work on Linux operating systems can install Singularity directly on their machines. But Windows and Mac OS users need to rely on a Linux virtual machine since Singularity only supports Linux. SyLabs, which has developed Singularity, provides Vagrant images (boxes) that are pre-installed Singularity on Linux. We recommend you install Vagrant. For the installation, please follow the guidelines from the official website: <a href="https://docs.sylabs.io/guides/3.8/admin-guide/installation.html" target="_blank">https://docs.sylabs.io/guides/3.8/admin-guide/installation.html</a></p><p dir="ltr">Note that we used Singularity 3.8 CE version.<br></p><p><br></p><h2><b>Getting Started with MOTIF</b></h2><h3><b>Preparing working directory</b></h3><p dir="ltr">Download MOTIF.tar and extract them into a working directory</p><ul><li>$ wget -O MOTIF.tar<a href="https://figshare.com/ndownloader/files/42023910" rel="noreferrer" target="_blank"> </a><a href="https://figshare.com/ndownloader/files/42024447" rel="noreferrer" target="_blank">https://figshare.com/ndownloader/files/42024447</a></li><li>$ tar xf MOTIF.tar</li><li>$ cd MOTIF</li></ul><p dir="ltr">Download Singularity image</p><ul><li>$ wget -O containers/motif_default.sif <a href="https://figshare.com/ndownloader/files/41974680" rel="noreferrer" target="_blank">https://figshare.com/ndownloader/files/41974680</a></li></ul><p dir="ltr">Download AFL++ (will make AFL++ directory in the working directory)</p><ul><li>$ wget -O AFL++-4.05a.tar <a href="https://figshare.com/ndownloader/files/40299817" rel="noreferrer" target="_blank">https://figshare.com/ndownloader/files/40299817</a></li><li>$ tar xf AFL++-4.05a.tar</li></ul><p dir="ltr">Download the subject MLFS</p><ul><li>$ wget -O case_studies/MLFS.tar <a href="https://figshare.com/ndownloader/files/41974686" rel="noreferrer" target="_blank">https://figshare.com/ndownloader/files/41974686</a></li><li>$ tar xf case_studies/MLFS.tar -C case_studies/</li></ul><p dir="ltr">Download the subject ASN1</p><ul><li>$ wget -O case_studies/ASN1.tar <a href="https://figshare.com/ndownloader/files/41974683" rel="noreferrer" target="_blank">https://figshare.com/ndownloader/files/41974683</a></li><li>$ tar xf case_studies/ASN1.tar -C case_studies/</li></ul><h3><b>Connecting to a vagrant box (for Windows or Mac OS users)</b></h3><p dir="ltr">The command below creates a virtual machine instance according to the Vagrantfile in the root repository. This will automatically bind the root repository to the directory /vagrant inside of the vagrant instance and connect to the vagrant instance:</p><ul><li>$ vagrant up && vagrant ssh</li></ul><p dir="ltr">Move to the bound directory, which is sharing between the vagrant instance and the host OS:</p><ul><li>[vagrant]$ cd /vagrant</li></ul><h3><b>Executing MOTIF with each subject</b></h3><p dir="ltr">By executing run_list.py, you can do mutation testing for all the mutants that are listed in a file. The following are the example commands for the target subjects.</p><ul><li>$ ./run_list.py -c case_studies/MLFS/config-mlfs.py --singularity -J _exp1 --timeout 600 case_studies/MLFS/live_mutants all</li><li>$ ./run_list.py -c case_studies/ASN1/config-asn1.py --singularity -J _exp1 --timeout 600 case_studies/ASN1/live_mutants all</li></ul><p dir="ltr">You can find the results in the directories `case_studies/MLFS/_exp1` and `case_studies/ASN1/_exp1`.</p><p dir="ltr">For more detail, please take a look at the guideline (README.md) in the MOTIF.tar.<br></p><p><br></p><h2><b>Getting Started with SEMuP</b></h2><h3><b>Preparing working directory</b></h3><p dir="ltr">Download MOTIF.tar and extract them into a working directory</p><ul><li>$ wget -O SEMUP.tar <a href="https://figshare.com/ndownloader/files/42023913" rel="noreferrer" target="_blank">https://figshare.com/ndownloader/files/42023913</a></li><li>$ tar xf SEMUP.tar</li><li>$ cd SEMUP</li></ul><p dir="ltr">Download singularity image</p><ul><li>$ wget -O containers/semup_full.sif <a href="https://figshare.com/ndownloader/files/40458779" rel="noreferrer" target="_blank">https://figshare.com/ndownloader/files/40458779</a></li></ul><p dir="ltr">Download the subject ASN1</p><ul><li>$ wget -O case_studies/ASN1_SEMuP.tar <a href="https://figshare.com/ndownloader/files/41974677" rel="noreferrer" target="_blank">https://figshare.com/ndownloader/files/41974677</a></li><li>$ tar xf case_studies/ASN1_SEMuP.tar -C case_studies/</li></ul><h3><b>Connecting to a vagrant box (for Windows or Mac OS users)</b></h3><p dir="ltr">We also provide a separate vagrant box for the SEMuP. Using the following commands, users can create the vagrant instance and connect. Note that /vagrant is the shared directory with the host OS.</p><ul><li>$ vagrant up & vagrant ssh</li><li>[vagrant]$ cd /vagrant</li></ul><h3><b>Executing SEMuP with each subject</b></h3><p dir="ltr">By executing `run_local.sh`, you can do mutation testing for all the mutants that are listed in a file. The following are the example commands for the target subject.</p><ul><li>$ cd case_studies/ASN1</li><li>case_studies/ASN1$ scripts/run_local.sh presemu ./WORKSPACE/DOWNLOADED/live_mutants ./WORKSPACE/_expLocal</li></ul><p dir="ltr">You can find the results in the directory `./WORKSPACE/_expLocal`. Note that each line in the `live_mutants` file will be a mutant ID in the result directory.</p><p dir="ltr">For more detail, please take a look at the guideline (README.md) in the SEMuP.tar.</p><p><br></p><p><br></p><p dir="ltr"><b>Acknowledgment</b></p><p dir="ltr">This research was supported by ESA via a GSTP element contract (RFQ/3-17554/21/NL/AS/kkIMPROVE) and by the NSERC Discovery and Canada Research Chair programs. The authors would like to thank Thierry Titcheu Chekam to help with the development of the SEMUs pipeline.</p>