Fuzzing for CPS Mutation Testing

This is the replication package for the paper `Fuzzing for CPS Mutation Testing`, which is accepted by ASE 2023.

Abstract

Mutation testing can help reduce the risks of releasing faulty software. For such reason, it is a desired practice for the development of embedded software running in safety-critical cyber-physical systems (CPS). Unfortunately, state-of-the-art test data generation techniques for mutation testing of C and C++ software, two typical languages for CPS software, rely on symbolic execution, whose limitations often prevent its application (e.g., it cannot test black-box components).

We propose a mutation testing approach that leverages fuzz testing, which has proved effective with C and C++ software. Fuzz testing automatically generates diverse test inputs that exercise program branches in a varied number of ways and, therefore, exercise statements in different program states, thus maximizing the likelihood of killing mutants, our objective.

We performed an empirical assessment of our approach with software components used in satellite systems currently in orbit. Our empirical evaluation shows that mutation testing based on fuzz testing kills a significantly higher proportion of live mutants than symbolic execution (i.e., up to an additional 47 percentage points). Further, when symbolic execution cannot be applied, fuzz testing provides significant benefits (i.e., up to 41% mutants killed). Our study is the first one comparing fuzz testing and symbolic execution for mutation testing; our results provide guidance towards the development of fuzz testing tools dedicated to mutation testing.

Package description

This replication package contains all the source code for MOTIF and SEMuP and singularity images containing dependencies that we used for our experiments. The MOTIF singularity image does not include AFL++ so that we can replace it without rebuilding the image. Additionally, we provide case study packages for each subject where it is open-source. The following list is the description of each file in this package:

• MOTIF.tar : all the source codes for the MOTIF pipeline
• motif_default.sif : singularity image that we used for our experiments
• AFL++4.05a.tar : the version of AFL we used for our experiments
• MLFS.tar : case study package of MLFS for MOTIF
• ASN1.tar : case study package of ASN1 for MOTIF
• SEMUP.tar : all the source codes for the SEMuP pipeline
• semup_full.sif : singularity image that we used for our experiments
• ASN1_SEMuP.tar : case study package of ASN1 for SEMuP
• results.tar : experiment results that are used in the paper

Note that there are additional packages:

• ASN1_20231024.tar: case study package of ASN1 for the updated version of MOTIF (see git repository)
• MLFS_20231024.tar: case study package of MLFS for the updated version of MOTIF (see git repository)
• AFL++-4.09a-Ubuntu22.04.tar: compiled AFL++ for Ubuntu 22.04
• motif_default_22.04.sif: singularity image for execution of MOTIF on Ubuntu 22.04

Pre-requisition

We use Singularity to provide the same environment for all the users. Users who work on Linux operating systems can install Singularity directly on their machines. But Windows and Mac OS users need to rely on a Linux virtual machine since Singularity only supports Linux. SyLabs, which has developed Singularity, provides Vagrant images (boxes) that are pre-installed Singularity on Linux. We recommend you install Vagrant. For the installation, please follow the guidelines from the official website: https://docs.sylabs.io/guides/3.8/admin-guide/installation.html

Note that we used Singularity 3.8 CE version.

Getting Started with MOTIF

Preparing working directory

Download MOTIF.tar and extract them into a working directory

• $ wget -O MOTIF.tar https://figshare.com/ndownloader/files/42024447
• $ tar xf MOTIF.tar
• $ cd MOTIF

Download Singularity image

• $ wget -O containers/motif_default.sif https://figshare.com/ndownloader/files/41974680

Download AFL++ (will make AFL++ directory in the working directory)

• $ wget -O AFL++-4.05a.tar https://figshare.com/ndownloader/files/40299817
• $ tar xf AFL++-4.05a.tar

Download the subject MLFS

• $ wget -O case_studies/MLFS.tar https://figshare.com/ndownloader/files/41974686
• $ tar xf case_studies/MLFS.tar -C case_studies/

Download the subject ASN1

• $ wget -O case_studies/ASN1.tar https://figshare.com/ndownloader/files/41974683
• $ tar xf case_studies/ASN1.tar -C case_studies/

Connecting to a vagrant box (for Windows or Mac OS users)

The command below creates a virtual machine instance according to the Vagrantfile in the root repository. This will automatically bind the root repository to the directory /vagrant inside of the vagrant instance and connect to the vagrant instance:

• $ vagrant up && vagrant ssh

Move to the bound directory, which is sharing between the vagrant instance and the host OS:

• [vagrant]$ cd /vagrant

Executing MOTIF with each subject

By executing run_list.py, you can do mutation testing for all the mutants that are listed in a file. The following are the example commands for the target subjects.

• $ ./run_list.py -c case_studies/MLFS/config-mlfs.py --singularity -J _exp1 --timeout 600 case_studies/MLFS/live_mutants all
• $ ./run_list.py -c case_studies/ASN1/config-asn1.py --singularity -J _exp1 --timeout 600 case_studies/ASN1/live_mutants all

You can find the results in the directories `case_studies/MLFS/_exp1` and `case_studies/ASN1/_exp1`.

For more detail, please take a look at the guideline (README.md) in the MOTIF.tar.

Getting Started with SEMuP

Preparing working directory

Download MOTIF.tar and extract them into a working directory

• $ wget -O SEMUP.tar https://figshare.com/ndownloader/files/42023913
• $ tar xf SEMUP.tar
• $ cd SEMUP

Download singularity image

• $ wget -O containers/semup_full.sif https://figshare.com/ndownloader/files/40458779

Download the subject ASN1

• $ wget -O case_studies/ASN1_SEMuP.tar https://figshare.com/ndownloader/files/41974677
• $ tar xf case_studies/ASN1_SEMuP.tar -C case_studies/

Connecting to a vagrant box (for Windows or Mac OS users)

We also provide a separate vagrant box for the SEMuP. Using the following commands, users can create the vagrant instance and connect. Note that /vagrant is the shared directory with the host OS.

• $ vagrant up & vagrant ssh
• [vagrant]$ cd /vagrant

Executing SEMuP with each subject

By executing `run_local.sh`, you can do mutation testing for all the mutants that are listed in a file. The following are the example commands for the target subject.

• $ cd case_studies/ASN1
• case_studies/ASN1$ scripts/run_local.sh presemu ./WORKSPACE/DOWNLOADED/live_mutants ./WORKSPACE/_expLocal

You can find the results in the directory `./WORKSPACE/_expLocal`. Note that each line in the `live_mutants` file will be a mutant ID in the result directory.

For more detail, please take a look at the guideline (README.md) in the SEMuP.tar.

Acknowledgment

This research was supported by ESA via a GSTP element contract (RFQ/3-17554/21/NL/AS/kkIMPROVE) and by the NSERC Discovery and Canada Research Chair programs. The authors would like to thank Thierry Titcheu Chekam to help with the development of the SEMUs pipeline.