3 files

Mitigating the Effects of Flaky Tests on Mutation Testing (Dataset and Tool)

posted on 04.06.2019, 20:35 by August Shi, Jonathan Bell, Darko Marinov
Mutation testing is widely used in research as a metric for evaluating the quality of test suites. Mutation testing runs the test suite on generated mutants (variants of the code under test), where a test suite kills a mutant if any of the tests fail when run on the mutant. Mutation testing implicitly assumes that tests exhibit deterministic behavior, in terms of their coverage and the outcome of a test (not) killing a certain mutant. Such an assumption does not hold in the presence of flaky tests, whose outcomes can non-deterministically differ even when run on the same code under test. Without reliable test outcomes, mutation testing can result in unreliable results, e.g., in our experiments, mutation scores vary by four percentage points on average between repeated executions, and 9% of mutant-test pairs have an unknown status. Many modern software projects suffer from flaky tests. We propose techniques that manage flakiness throughout the mutation testing process, largely based on strategically re-running tests. We implement our techniques by modifying the open-source mutation testing tool, PIT. Our evaluation on 30 projects shows that our techniques reduce the number of "unknown" (flaky) mutants by 79.4%.

This artifact contains the dataset and tool that accompany this ISSTA 2019 paper.


NSF CCF-1421503

NSF CNS-1646305

NSF CNS-1740916

NSF CCF-1763788

NSF CCF-1763822

NSF OAC-1839010