Supporting Error Chains in Static Analysis for Precise Evaluation Results and Enhanced Usability

Abstract

Context: Static analyses are well-established to aid in understanding bugs or vulnerabilities during the development process or in large-scale studies. A low false positive rate is essential for the adaption in practice and for precise results of empirical studies. Unfortunately, static analyses tend to report where a vulnerability manifests rather than the fix location. This can cause presumed false positives or imprecise results. Method: To address this problem, we designed an adaption of an existing static analysis algorithm that can distinguish between a manifestation and fix location and reports error chains. Each error chain presents the dependency between the fix location with at least one manifestation location. We used our tool for a case study of 471 GitHub repositories and conducted an expert interview to investigate usability implications of the change. Further, we benchmarked both analysis versions to compare the runtime impact. Result: We found that 50% of the projects with a report had at least one error chain. During our expert interview, all participants required fewer executions of the static analysis if they used our adapted version. Our performance benchmark demonstrated that our improvement caused only a minimal runtime overhead of less than 4%. Conclusion: Our results indicate that error chains occur frequently in real-world projects and ignoring them can lead to imprecise evaluation results. The performance benchmark indicates that our tool is a feasible and efficient solution for detecting error-chains in real-world projects. Further, our results indicate that the usability of static analyses benefits from supporting error chains.

Data

This artefact contains additional information for our evaluation.

Folder code_study (RQ1)

• The folder JavaCryptographicAchitecture_BET contains the CrySL rules for the JCA that we used for the code study.
• The file SUBS.jar is the version of SUBS that we used for our code study.
• The file README.md describes how to use the Docker image for scanning the code with CogniCryptSUBS.
• The file CREDENTIALS.txt is a dummy file for the GitHub tokens required for the analysis.
• The file run_cc_subs.sh is a helper script to execute CogniCryptSUBS and used by the Docker container.
• The file `Dockerfile` is the Docker image used for the code study.

Folder performance_analysis (RQ2)

• The folder 1_run_performance_analysis/JavaCryptographicArchitecture contains the CrySL rules for the JCA that we used for the benchmark that do not support Backward Error Tracking (BET).
• The folder 1_run_performance_analysis/JavaCryptographicArchitecture_BET contains the CrySL rules for the JCA that we used for the benchmark that support BET.
• The different 1_run_performance_analysis/*.jar files are the different evaluated versions of CogniCrypt and CogniCrypt_SUBS.
• The file 1_run_performance_analysis/Dockerfile is the Docker image used to execute the benchmark.
• The file 1_run_performance_analysis/run_performance_analysis.sh includes the commands to execute the different tools on our benchmark and the different target folders for the different configurations/groups of the benchmark.
• The folder 2_parse_results/data contains the results obtained for the five different configurations for the different tools.
• The file 2_parse_results/generate_graphics.py generates the graphics used in the paper.
• The folder results contains the graphics, such as Fig. 4, for the different configurations.

Folder expert_interview (RQ3)

• The code examples for task 1 and 2 are in the folder expertinterview_examplecode1 and expert interview_examplecode2, respectively.
• The invitation and questions are in the file `expert interview.md`.
• An overview of the obtained results are in the file `expert interview_results.csv`.

Further, we include the graphics for the runtime evaluation as pdf-files.

Changes

• Version 2: Restructure the main folder to include one folder for each research question answered in the paper. Further, added data for the code study and more details for the performance benchmark.
• Version 1: Add details for the expert interview and pdf-files for the performance benchmark. All files were added to the main folder.