Artifact for “Analysis of non-Markovian repairable fault trees through rare event simulation”

by C.E. Budde, P.R. D'Argenio, R.E. Monti, and M. Stoelinga.

Minimal instructions

Copy the artifact/ directory into a Debian-based Linux system (that has build-essentials and cmake installed), and execute the script run_artifact.sh:

cd artifact; ./run_artifact.sh

NOTE: If working with virtual machines and shared directorys, please copy the archive containing the artifact into a directory inside of the machine and extract it there; otherwise (e.g. if working directly in a shared directory) symbolic links contained in the archive may not work correctly.

By default this artifact performs a so-called DEMO run, that takes between 6h and 24h of wallclock run time in a single machine. FULL experimental reproduction requires a PBS-administered cluster: instructions are included at the end of this file.

Contents of the artifact

• This README file
• The LICENSE file with the distribution license of the artifact
• The main execution script
  • file run_artifact.sh
• Installation utilities for the tool chain
  • in directory setup/
• All models and scripts for experimental reproduction
  • in directory experiments/
• All files of the results reported in the paper
  • in directory paper_experiments_results/

Environment and requirements

The artifact is prepared to run in Debian-based Linux distributions; it has been tested in Ubuntu 20.04. The GNU essentials for C++ compilation must be available, i.e. the GNU compiler collection (gcc, g++) and Cmake version 2.8.13 or above.

You can download a VirtualBox VM image that satisfies all these requirements from: https://zenodo.org/record/4041464#.X5GeOyVS-EJ . We refer to this virtual machine in its standard configuration with 1 processor and 2 GB RAM as the TACAS VM.

Note that experimental runtimes depend strongly on the hardware and virtualisation employed. These experiments use wallclock runtime as simulation budget, so lower machine performance translates into wider (absolute) confidence intervals widths. In particular, to create the plots in figures 7 through 11 in the paper, the CI widths were obtained from executions in the (non-virtualised) nodes of a cluster with CPUs Intel Xeon E5-2680 v4 @ 2.40 GHz (14 cores, 35M cache), each with 384 GB of DDR4 RAM @ 1600 MHz. Executions in the TACAS VM will produce CIs whose absolute width is considerably higher than those used for the plots presented in the paper. Instead, if the FULL (as opposed to the DEMO) run is executed, the relative CI width among the different run types should remain comparable to those reported in figures 7 through 11. There are instructions on how to do this a the end of this file.

Files for inspection

All files included in this artifact are freely available for inspection, e.g. the source code of FIG, and the Kepler compiler to IOSA. That said, manual inspection is probably mostly relevant for the results reported in the paper.

The directory paper_experiments_results/ contains the full output of the tool executions (in a PBS-administered cluster), that was used to create the plots for figures 7 through 11 in the paper. The following files and locations in that directory may be of prime interest:

• plots/*rel.pdf are the bar plots for our un-reliability studies
• plots/*ava.pdf are the bar plots for our un-availability studies
• plots/scatterplots.pdf includes the scatter plots of all experiments
• data/summary.csv is the CSV file used to creater such scatter plots
• summaries/ contains all CI widths achieved by every individual run (prior to aggregation), split per model instance

Another interesting directory is experiments/models/, which contains the fault tree models used for experimentation. This is all part of our tool chain:

• experiments/models/FT/*.ft are the fault tree models written in the Kepler textual format
• experiments/models/IOSA/*.sa are the corresponding IOSA models
• experiments/models/README.md explains how to generate the latter from the former with our tool chain

Experiment reproduction

Running this artifact in its default configuration will perform a DEMO run, that minimally demonstrates the results presented in the paper. The entry point is the executable script run_artifact.sh. To run the artifact, open a terminal, move into the artifact/ directory (where this README file resides), and execute the script:

cd artifact/; ./run_artifact.sh

There are two running modes, between which the user can choose by setting the proper value to the variable DEMO_RUN in line 11 of run_artifact.sh:

• The DEMO run, chosen by setting DEMO_RUN=1, performs a reduced number of runs in a reduced set of models
  • this is the default configuration of the artifact
  • in a direct execution on a PC with an Intel Xeon E-2124G CPU, this takes around 6 h
  • in the TACAS VM this can take approximately one full day
• The FULL run, chosen by setting DEMO_RUN=0, replicates all experiments of the paper
  • this must be run in a cluster with the PBS job manager (or SLURM, but this requires adapting the scripts by hand)
  • you will probably need to set the variables PBS_QUEUE and PBS_JOB_LIMIT (in lines 16 and 17 of scripts/run_full.sh) to match your cluster configuration
  • with a limit of 30 parallel jobs per user, this took approximately 2 days to run

Execution steps of the artifact

For both run types, executing the artifact will perform the following steps:

1. Install the required dependencies
   • deb & python3 packages were chosen to complete the needs of the TACAS VM
   • installation requires root privileges: please enter the password when prompted
   • if you are running this in the TACAS VM, the password is “tacas21”
   • if a working installation of gnuplot is found, this step is omitted
2. Install the FIG tool in the environment
   • installation requires root privileges
   • external dependencies might be needed: agree to installing these if prompted
   • if a working installation of FIG is found, this step is omitted
3. Perform the experimental replication:
   • execute experiments/run.sh in the chosen mode (DEMO or FULL)
   • the above includes summary and plotting, to better visualise the results

In Step 1, most packages are required for plotting purposes only, but a few like libbison-dev and libf-dev are needed by the FIG tool.

Step 2 is crucial: if installation fails, please refer to the FIG homepage for manual installation. In such case, we recommend to download and install the RFT_journal tagged version of FIG.

Step 3 runs the FIG tool on the models presented in the paper. They all build confidence intervals using Student’s-T quantiles that estimate the requested transient or steady-state metrics. Each experiment is repeated N times (N=4 for DEMO and 10 for FULL), running for a simulation time bound chosen per experiment. All values can be read in the bash script files experiments/scripts/run_{demo,full}.sh.

Troubleshooting artifact setup in the TACAS VM

If you are running this artifact in the TACAS VM, a daemon for unattended upgrades may lock the package manager on startup. This interferes with the installation of deb packages, required to satisfy the dependencies of the FIG tool.

Thus, if during installation you see the following error message: E: Could not get lock /var/lib/apt/lists ... then you will have to manually kill the “unattended-upgr” process prior to executing the artifact. Note that this requires root privileges, and you may need to do it twice in a row!

Results inspection

When all experiments finalise, the artifact prints the location of the results, e.g. All plots ready in /home/tacas21/artifact/experiments/results/2021_10_29-13_45_56/plots for the TACAS VM run on Oct 29 (2021) at 13:45:56. In this example let DIR=/home/tacas21/artifact/experiments/results/2021_10_29-13_45_56/, then the following files and locations are of prime interest:

1. DIR/data/summary.csv has the values from which the scatter plots are drawn (Sec. 6.2 in the paper)
   • Consider using the scripts/scatterplots.tex LaTeX file to create the corresponding scatter plots
   • Although automatic, this is not performed by default in the artifact because it would require to install the texlive package, which would bloat the artifact size to gigabytes.
2. DIR/plots/*{rel,ava}.svg are respectively the bar plots for un-{reli,avail}ability studies (Sec. 6.3 in the paper)
   • If desired, svg files can be converted to pdf with tools such as svg2pdf
   • In any case, the eog command of Ubuntu can open svg files just as well (e.g. ~$ eog DIR/plots/DSPARE_3_rel.svg)
3. DIR/summaries/ contains all CI widths achieved by every individual run, prior to aggregation, split per model instance
   • These are plain ASCII text, actually meant as a middle-product used for plotting
   • We mention these files as they are the last stage where the raw numbers produced by FIG can be seen (namely the width of confidence intervals for 95% confidence level), before its aggregation into robust means and standard deviations.

On the reproducibility of exact results

The FIG tool uses the <random> header from the C++ STL:

• the RNG used is a 64-bit Mersenne Twister (std::mt19937_64);
• to run stochastic simulations, FIG uses this RNG to sample values from distributions such as exponential, Erlang, etc., also provided by the C++ STL;
• although the Mersenne Twister RNG is deterministic for a given seed, and its behaviour is fully specified by the C++ standard, the distributions are not (see e.g. https://stackoverflow.com/q/48730363);
• so even for a fixed RNG seed, the rare-event values estimated by FIG will almost certainly not match exactly the numbers obtained during our experiments for the paper (this was corroborated when running FIG in the TACAS VM).
• still, the general trend reported in Sec. 6 of the paper should be observed: the more repetitions performed per experiment, the closer the results will resemble figures 7 through 11 in the paper.