FuzzBench: 2020-05-24 report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggreage (cross-benchmark) ranking of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached covereges (lower value is better).
By avg. score
average normalized score
fuzzer
honggfuzz 96.60
symqemu 91.87
entropic 91.74
lafintel 90.32
aflsmart 90.24
afl 90.01
aflplusplus 88.83
aflfast 86.88
fastcgs_lm 86.10
mopt 85.84
libfuzzer 85.32
fairfuzz 81.35
eclipser 70.49
By avg. rank
average rank
fuzzer
honggfuzz 3.76
symqemu 4.07
afl 4.74
aflsmart 4.93
entropic 5.52
fastcgs_lm 5.76
mopt 6.79
aflplusplus 7.57
libfuzzer 8.74
lafintel 8.81
aflfast 8.90
fairfuzz 9.26
eclipser 12.14
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median coverages on each benchmark
    fuzzer afl aflfast aflplusplus aflsmart eclipser entropic fairfuzz fastcgs_lm honggfuzz lafintel libfuzzer mopt symqemu
    benchmark
    bloaty_fuzz_target 5374.0 5200.0 5315.0 5355.0 4148.0 4785.0 5191.0 5704.0 5531.0 5074.0 4531.0 5666.0 5305.0
    curl_curl_fuzzer_http 5414.0 5298.0 5331.0 5410.0 4419.0 4995.0 4754.0 5411.0 5394.0 5330.0 4795.0 5396.0 5395.0
    freetype2-2017 5346.0 5237.0 5152.0 5309.0 4430.0 5614.0 5333.0 5335.0 6856.0 5086.0 4198.0 5333.0 6256.0
    harfbuzz-1.3.2 4260.0 4179.0 4182.0 4262.0 3364.0 4197.0 3638.0 4252.0 4376.0 4109.0 4082.0 4248.0 4310.0
    jsoncpp_jsoncpp_fuzzer 634.0 634.0 634.0 634.0 590.0 635.0 634.0 634.0 635.0 630.0 635.0 634.0 634.0
    lcms-2017-03-21 1146.0 904.0 1163.0 1157.0 460.0 1323.0 1182.0 852.0 999.0 1128.0 1216.0 861.0 1425.0
    libjpeg-turbo-07-2017 1438.0 1431.0 1441.0 1433.0 1096.0 1449.0 1110.0 1448.0 1432.0 1425.0 1296.0 1435.0 1452.0
    libpcap_fuzz_both 21.0 21.0 18.0 18.0 887.0 1709.0 21.0 20.0 2018.0 1556.0 1583.0 21.0 22.0
    libpng-1.2.56 631.0 627.0 630.0 673.0 503.0 641.0 631.0 524.0 676.0 643.0 624.0 525.0 676.0
    libxml2-v2.9.2 4586.0 4427.0 4403.0 4681.0 1667.0 4491.0 3419.0 2219.0 4570.0 4411.0 4276.0 2402.0 4541.0
    mbedtls_fuzz_dtlsclient 1677.0 1601.0 1657.0 1673.0 1417.0 1631.0 1662.0 1680.0 1678.0 1597.0 1556.0 1681.0 1662.0
    openssl_x509 4076.0 4072.0 4076.0 4076.0 4056.0 4077.0 4055.0 4077.0 4070.0 4071.0 4069.0 4075.0 4076.0
    openthread-2019-12-23 1669.0 1479.0 1661.0 1674.0 1618.0 1490.0 1091.0 1669.0 1687.0 1473.0 1487.0 1672.0 1683.0
    php_php-fuzz-parser 11219.0 11114.0 11114.0 11242.0 9893.0 11064.0 10798.0 11331.0 11503.0 10920.0 10235.0 11205.0 11272.0
    proj4-2017-08-14 2142.0 2025.0 2067.0 2108.0 154.0 2352.0 2044.0 1966.0 3260.0 2088.0 2257.0 1976.0 2134.0
    re2-2014-12-09 2248.0 2246.0 2241.0 2249.0 1935.0 2278.0 2247.0 2231.0 2272.0 2230.0 2287.0 2228.0 2262.0
    sqlite3_ossfuzz 17305.0 16653.0 16577.0 17210.0 5403.5 11162.0 9391.0 17690.0 12687.0 11884.0 8802.0 17279.0 16730.0
    systemd_fuzz-link-parser 991.0 987.0 989.0 990.0 934.0 987.0 895.0 990.0 1009.0 980.0 776.0 990.0 990.0
    vorbis-2017-12-11 1007.0 1001.0 1006.0 1015.0 877.0 1005.0 1005.0 1016.0 1003.0 984.0 792.0 1015.0 1009.0
    woff2-2016-05-06 1065.0 960.0 1010.0 1030.0 804.0 1026.0 945.0 1075.0 1090.0 1122.0 1000.0 1021.0 1089.0
    zlib_zlib_uncompress_fuzzer 330.0 326.0 329.0 333.0 313.0 334.0 331.0 330.0 331.0 329.0 334.0 329.0 330.0

bloaty_fuzz_target summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fastcgs_lm 86400 15.0 5691.066667 58.698341 5596.0 5644.5 5704.0 5741.0 5765.0
    mopt 86400 15.0 5669.933333 57.868653 5567.0 5634.0 5666.0 5707.0 5766.0
    honggfuzz 86400 15.0 5520.266667 50.374975 5416.0 5492.5 5531.0 5553.0 5600.0
    afl 86400 15.0 5412.666667 151.727983 5177.0 5300.5 5374.0 5508.0 5656.0
    aflsmart 86400 15.0 5384.400000 157.189967 5144.0 5283.5 5355.0 5430.5 5676.0
    aflplusplus 86400 15.0 5300.266667 103.235145 5096.0 5256.0 5315.0 5358.0 5449.0
    symqemu 86400 15.0 5316.533333 116.028240 5105.0 5268.5 5305.0 5388.5 5602.0
    aflfast 86400 15.0 5234.266667 165.646554 5022.0 5096.5 5200.0 5353.0 5537.0
    fairfuzz 86400 15.0 5124.466667 312.635083 4651.0 4800.5 5191.0 5305.0 5656.0
    lafintel 86400 15.0 5026.266667 104.154738 4812.0 4930.0 5074.0 5095.5 5153.0
    entropic 86400 15.0 4783.533333 121.463614 4590.0 4722.5 4785.0 4850.5 4969.0
    libfuzzer 86400 15.0 4431.200000 242.237546 3982.0 4332.0 4531.0 4616.0 4642.0
    eclipser 86400 15.0 4179.666667 130.986186 4021.0 4099.0 4148.0 4222.0 4525.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

curl_curl_fuzzer_http summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 86400 15.0 5423.266667 34.810644 5388.0 5403.0 5414.0 5424.5 5510.0
    fastcgs_lm 86400 15.0 5415.733333 14.916274 5393.0 5406.5 5411.0 5425.0 5445.0
    aflsmart 86400 15.0 5415.133333 31.297802 5367.0 5402.5 5410.0 5415.5 5489.0
    mopt 86400 15.0 5401.266667 29.841167 5357.0 5386.5 5396.0 5413.5 5475.0
    symqemu 86400 15.0 5400.666667 26.515943 5356.0 5384.5 5395.0 5413.0 5459.0
    honggfuzz 86400 15.0 5388.400000 82.423818 5230.0 5333.0 5394.0 5439.5 5520.0
    aflplusplus 86400 15.0 5333.666667 26.332128 5295.0 5316.0 5331.0 5351.0 5392.0
    lafintel 86400 15.0 5334.333333 25.061828 5295.0 5320.0 5330.0 5346.0 5380.0
    aflfast 86400 15.0 5301.200000 45.066934 5234.0 5269.5 5298.0 5317.5 5411.0
    entropic 86400 15.0 4992.000000 209.877856 4477.0 4947.5 4995.0 5113.0 5272.0
    libfuzzer 86400 15.0 4746.933333 161.128816 4416.0 4682.5 4795.0 4844.0 4972.0
    fairfuzz 86400 15.0 4712.533333 272.909160 4189.0 4579.0 4754.0 4849.5 5319.0
    eclipser 86400 15.0 4413.733333 37.434641 4312.0 4410.5 4419.0 4435.0 4460.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2-2017 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 86400 15.0 6877.066667 259.216530 6390.0 6704.0 6856.0 7017.0 7349.0
    symqemu 86400 15.0 6227.000000 103.682896 5965.0 6157.5 6256.0 6304.5 6370.0
    entropic 86400 15.0 5609.800000 193.645184 5167.0 5570.0 5614.0 5639.0 6009.0
    afl 86400 15.0 5346.400000 39.138582 5284.0 5322.0 5346.0 5375.5 5424.0
    fastcgs_lm 86400 15.0 5343.200000 39.204956 5304.0 5315.5 5335.0 5356.5 5458.0
    fairfuzz 86400 15.0 5324.066667 42.849348 5231.0 5301.0 5333.0 5351.5 5398.0
    mopt 86400 15.0 5327.733333 52.004762 5223.0 5295.5 5333.0 5377.0 5394.0
    aflsmart 86400 15.0 5307.800000 72.395935 5178.0 5277.0 5309.0 5323.5 5510.0
    aflfast 86400 15.0 5209.466667 54.599669 5100.0 5162.0 5237.0 5246.0 5271.0
    aflplusplus 86400 15.0 5139.266667 55.885683 5056.0 5089.5 5152.0 5177.0 5230.0
    lafintel 86400 15.0 5096.066667 49.913449 5001.0 5071.0 5086.0 5120.0 5195.0
    eclipser 86400 15.0 4382.600000 222.142104 4029.0 4188.0 4430.0 4567.5 4694.0
    libfuzzer 86400 15.0 4382.266667 417.471379 3918.0 4123.0 4198.0 4626.5 5172.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz-1.3.2 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 86400 15.0 4378.400000 57.334607 4258.0 4352.0 4376.0 4420.5 4473.0
    symqemu 86400 15.0 4309.133333 39.142353 4248.0 4293.5 4310.0 4316.0 4419.0
    aflsmart 86400 15.0 4274.466667 50.111115 4228.0 4246.5 4262.0 4276.5 4415.0
    afl 86400 15.0 4296.466667 85.598620 4234.0 4246.5 4260.0 4285.5 4504.0
    fastcgs_lm 86400 15.0 4266.533333 59.255942 4211.0 4239.0 4252.0 4272.5 4467.0
    mopt 86400 15.0 4279.933333 79.103428 4208.0 4224.5 4248.0 4325.0 4447.0
    entropic 86400 15.0 4196.933333 77.216270 4064.0 4140.5 4197.0 4225.5 4342.0
    aflplusplus 86400 15.0 4187.800000 38.116738 4149.0 4163.5 4182.0 4196.5 4306.0
    aflfast 86400 15.0 4189.333333 57.386741 4122.0 4144.5 4179.0 4197.5 4322.0
    lafintel 86400 15.0 4140.933333 83.084696 4039.0 4087.5 4109.0 4208.5 4349.0
    libfuzzer 86400 15.0 4068.800000 102.581118 3920.0 3976.0 4082.0 4119.0 4250.0
    fairfuzz 86400 15.0 3668.533333 239.291175 3403.0 3546.5 3638.0 3731.5 4344.0
    eclipser 86400 15.0 3368.600000 24.280504 3342.0 3350.0 3364.0 3382.0 3416.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 86400 15.0 634.933333 0.258199 634.0 635.0 635.0 635.0 635.0
    honggfuzz 86400 15.0 634.933333 0.258199 634.0 635.0 635.0 635.0 635.0
    libfuzzer 86400 15.0 634.933333 0.258199 634.0 635.0 635.0 635.0 635.0
    afl 86400 15.0 634.000000 0.000000 634.0 634.0 634.0 634.0 634.0
    aflfast 86400 15.0 633.800000 0.414039 633.0 634.0 634.0 634.0 634.0
    aflplusplus 86400 15.0 633.933333 0.258199 633.0 634.0 634.0 634.0 634.0
    aflsmart 86400 15.0 634.000000 0.000000 634.0 634.0 634.0 634.0 634.0
    fairfuzz 86400 15.0 634.000000 0.000000 634.0 634.0 634.0 634.0 634.0
    fastcgs_lm 86400 14.0 633.928571 0.267261 633.0 634.0 634.0 634.0 634.0
    mopt 86400 15.0 634.000000 0.000000 634.0 634.0 634.0 634.0 634.0
    symqemu 86400 15.0 633.933333 0.258199 633.0 634.0 634.0 634.0 634.0
    lafintel 86400 15.0 629.733333 0.798809 629.0 629.0 630.0 630.0 631.0
    eclipser 86400 15.0 590.933333 9.967566 566.0 587.0 590.0 596.0 608.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms-2017-03-21 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symqemu 86400 15.0 1445.866667 54.044779 1353.0 1410.0 1425.0 1497.5 1529.0
    entropic 86400 15.0 1320.666667 58.928124 1209.0 1271.5 1323.0 1374.5 1394.0
    libfuzzer 86400 15.0 1217.333333 51.262165 1127.0 1197.5 1216.0 1235.5 1352.0
    fairfuzz 86400 15.0 1167.266667 82.600992 978.0 1150.0 1182.0 1214.5 1316.0
    aflplusplus 86400 15.0 1106.000000 111.343741 917.0 1016.5 1163.0 1184.0 1267.0
    aflsmart 86400 15.0 1151.266667 58.993301 996.0 1112.5 1157.0 1191.0 1231.0
    afl 86400 15.0 1125.800000 103.075700 920.0 1092.5 1146.0 1191.0 1278.0
    lafintel 86400 15.0 1145.866667 124.602721 935.0 1102.5 1128.0 1274.0 1319.0
    honggfuzz 86400 15.0 1069.933333 150.714236 923.0 941.5 999.0 1233.5 1316.0
    aflfast 86400 15.0 857.133333 181.662508 399.0 874.5 904.0 914.0 1124.0
    mopt 86400 15.0 871.533333 29.963947 839.0 842.0 861.0 899.0 910.0
    fastcgs_lm 86400 15.0 870.133333 30.621810 839.0 843.5 852.0 902.5 911.0
    eclipser 86400 15.0 529.000000 162.464502 366.0 437.0 460.0 521.5 850.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo-07-2017 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symqemu 86400 15.0 1451.000000 13.021959 1434.0 1438.5 1452.0 1459.5 1472.0
    entropic 86400 15.0 1448.200000 6.624629 1433.0 1445.0 1449.0 1452.5 1455.0
    fastcgs_lm 86400 15.0 1424.800000 59.364732 1267.0 1435.5 1448.0 1453.5 1466.0
    aflplusplus 86400 15.0 1411.066667 71.671340 1272.0 1407.0 1441.0 1450.5 1473.0
    afl 86400 15.0 1443.733333 9.527904 1433.0 1437.5 1438.0 1453.5 1460.0
    mopt 86400 15.0 1418.800000 53.077033 1290.0 1431.0 1435.0 1439.0 1465.0
    aflsmart 86400 15.0 1396.133333 73.268456 1266.0 1359.5 1433.0 1437.0 1452.0
    honggfuzz 86400 15.0 1377.533333 77.912833 1281.0 1285.0 1432.0 1439.0 1453.0
    aflfast 86400 15.0 1398.266667 68.230143 1248.0 1398.0 1431.0 1436.0 1452.0
    lafintel 86400 15.0 1371.200000 84.733363 1237.0 1264.0 1425.0 1429.0 1446.0
    libfuzzer 86400 15.0 1259.133333 201.896319 801.0 1175.5 1296.0 1423.5 1427.0
    fairfuzz 86400 15.0 1120.000000 94.094178 929.0 1075.0 1110.0 1223.0 1231.0
    eclipser 86400 15.0 1079.000000 66.221490 912.0 1039.5 1096.0 1120.0 1177.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpcap_fuzz_both summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 86400 15.0 1993.000000 130.876168 1741.0 1933.0 2018.0 2107.0 2134.0
    entropic 86400 15.0 1723.266667 135.795259 1503.0 1648.5 1709.0 1824.5 1940.0
    libfuzzer 86400 15.0 1571.466667 74.013384 1463.0 1506.0 1583.0 1628.5 1693.0
    lafintel 86400 15.0 1618.666667 186.580380 1377.0 1486.0 1556.0 1725.0 2037.0
    eclipser 86400 15.0 961.066667 155.119524 826.0 831.5 887.0 1116.5 1213.0
    symqemu 86400 15.0 22.733333 1.579632 21.0 22.0 22.0 24.5 25.0
    afl 86400 15.0 20.866667 1.552264 18.0 21.0 21.0 21.0 25.0
    aflfast 86400 15.0 20.800000 1.264911 18.0 21.0 21.0 21.0 23.0
    fairfuzz 86400 15.0 28.200000 29.282369 18.0 21.0 21.0 21.0 134.0
    mopt 86400 15.0 33.800000 53.743039 18.0 18.0 21.0 21.0 228.0
    fastcgs_lm 86400 15.0 25.600000 24.473893 18.0 18.0 20.0 20.0 114.0
    aflplusplus 86400 15.0 19.400000 1.549193 18.0 18.0 18.0 21.0 21.0
    aflsmart 86400 15.0 18.600000 1.242118 18.0 18.0 18.0 18.0 21.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng-1.2.56 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 86400 15.0 677.333333 4.937707 666.0 674.5 676.0 681.5 685.0
    symqemu 86400 15.0 676.533333 7.120059 669.0 672.0 676.0 678.5 697.0
    aflsmart 86400 15.0 672.866667 3.181793 665.0 671.5 673.0 675.0 678.0
    lafintel 86400 15.0 642.866667 1.505545 640.0 642.0 643.0 644.0 645.0
    entropic 86400 15.0 641.266667 5.147353 632.0 637.5 641.0 645.0 649.0
    afl 86400 15.0 630.466667 0.990430 628.0 630.0 631.0 631.0 632.0
    fairfuzz 86400 15.0 631.200000 1.521278 628.0 630.0 631.0 632.5 633.0
    aflplusplus 86400 15.0 628.800000 2.858571 623.0 627.0 630.0 631.0 633.0
    aflfast 86400 15.0 627.000000 1.647509 624.0 626.0 627.0 628.0 630.0
    libfuzzer 86400 15.0 627.733333 6.649024 620.0 621.5 624.0 634.0 638.0
    mopt 86400 15.0 525.200000 2.007130 521.0 524.5 525.0 526.0 530.0
    fastcgs_lm 86400 15.0 524.466667 4.389381 519.0 523.0 524.0 525.0 538.0
    eclipser 86400 15.0 506.600000 18.511772 470.0 496.0 503.0 518.5 537.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2-v2.9.2 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflsmart 86400 15.0 4612.733333 186.289508 4292.0 4485.0 4681.0 4719.5 4957.0
    afl 86400 15.0 4553.466667 128.178373 4345.0 4445.0 4586.0 4651.0 4752.0
    honggfuzz 86400 15.0 4574.666667 83.975904 4388.0 4538.0 4570.0 4631.0 4711.0
    symqemu 86400 15.0 4551.666667 140.542656 4271.0 4438.5 4541.0 4667.5 4731.0
    entropic 86400 15.0 4492.200000 98.108686 4327.0 4425.5 4491.0 4544.5 4661.0
    aflfast 86400 15.0 4411.600000 200.048851 4103.0 4264.0 4427.0 4583.5 4680.0
    lafintel 86400 15.0 4375.400000 113.854418 4174.0 4302.0 4411.0 4439.5 4581.0
    aflplusplus 86400 15.0 4405.533333 156.103385 4143.0 4296.0 4403.0 4539.5 4586.0
    libfuzzer 86400 15.0 4325.133333 108.390080 4164.0 4238.5 4276.0 4430.5 4487.0
    fairfuzz 86400 15.0 3432.533333 366.880648 2695.0 3227.5 3419.0 3652.0 4063.0
    mopt 86400 15.0 2999.666667 788.010394 2206.0 2294.5 2402.0 3747.0 4045.0
    fastcgs_lm 86400 15.0 2596.533333 744.234013 2148.0 2176.5 2219.0 2461.5 4243.0
    eclipser 86400 15.0 1666.733333 24.496258 1615.0 1656.5 1667.0 1679.5 1714.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    mopt 86400 15.0 1724.733333 174.069554 1665.0 1674.5 1681.0 1686.0 2353.0
    fastcgs_lm 86400 15.0 1753.066667 201.822930 1665.0 1668.5 1680.0 1688.0 2307.0
    honggfuzz 86400 15.0 1676.466667 9.905746 1664.0 1669.0 1678.0 1679.0 1700.0
    afl 86400 15.0 1716.733333 170.955494 1651.0 1664.0 1677.0 1684.0 2333.0
    aflsmart 86400 15.0 1757.866667 215.883192 1664.0 1669.5 1673.0 1686.5 2290.0
    fairfuzz 86400 15.0 1681.933333 78.134012 1648.0 1650.5 1662.0 1676.0 1961.0
    symqemu 86400 15.0 1661.133333 13.362349 1634.0 1654.5 1662.0 1668.5 1688.0
    aflplusplus 86400 15.0 1690.266667 129.227632 1647.0 1653.5 1657.0 1662.0 2157.0
    entropic 86400 15.0 1633.200000 12.196018 1611.0 1626.0 1631.0 1642.0 1655.0
    aflfast 86400 15.0 1597.400000 11.836988 1572.0 1590.5 1601.0 1604.0 1615.0
    lafintel 86400 15.0 1595.133333 9.156315 1580.0 1587.5 1597.0 1601.5 1610.0
    libfuzzer 86400 15.0 1538.800000 79.562374 1413.0 1476.0 1556.0 1615.0 1626.0
    eclipser 86400 15.0 1414.400000 14.246804 1390.0 1405.5 1417.0 1422.0 1440.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 86400 15.0 4077.200000 1.521278 4075.0 4076.5 4077.0 4078.0 4081.0
    fastcgs_lm 86400 15.0 4077.333333 0.816497 4076.0 4077.0 4077.0 4078.0 4079.0
    afl 86400 15.0 4076.066667 1.032796 4075.0 4075.5 4076.0 4076.0 4079.0
    aflplusplus 86400 15.0 4076.266667 1.533747 4074.0 4075.0 4076.0 4078.0 4079.0
    aflsmart 86400 15.0 4076.266667 1.334523 4075.0 4075.0 4076.0 4077.0 4079.0
    symqemu 86400 15.0 4076.133333 1.407463 4074.0 4075.0 4076.0 4077.0 4079.0
    mopt 86400 15.0 4075.466667 0.833809 4074.0 4075.0 4075.0 4076.0 4077.0
    aflfast 86400 15.0 4071.933333 2.737743 4068.0 4070.5 4072.0 4074.0 4076.0
    lafintel 86400 15.0 4071.133333 3.090693 4066.0 4069.5 4071.0 4073.5 4075.0
    honggfuzz 86400 15.0 4069.400000 2.746426 4064.0 4068.5 4070.0 4071.5 4072.0
    libfuzzer 86400 15.0 4068.333333 2.609506 4062.0 4067.0 4069.0 4069.5 4073.0
    eclipser 86400 15.0 4053.933333 5.243590 4046.0 4049.0 4056.0 4057.5 4062.0
    fairfuzz 86400 15.0 4056.800000 10.331643 4044.0 4049.5 4055.0 4065.5 4073.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread-2019-12-23 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 86400 15.0 1673.066667 22.717101 1636.0 1649.5 1687.0 1690.0 1698.0
    symqemu 86400 15.0 1682.333333 11.437199 1660.0 1674.5 1683.0 1693.0 1700.0
    aflsmart 86400 15.0 1661.333333 58.616997 1457.0 1666.5 1674.0 1685.5 1703.0
    mopt 86400 15.0 1644.733333 72.417901 1463.0 1661.0 1672.0 1676.0 1695.0
    afl 86400 15.0 1674.200000 14.423196 1655.0 1663.0 1669.0 1684.0 1701.0
    fastcgs_lm 86400 15.0 1656.133333 53.954831 1463.0 1665.5 1669.0 1673.0 1685.0
    aflplusplus 86400 15.0 1599.800000 95.337146 1460.0 1474.5 1661.0 1667.5 1687.0
    eclipser 86400 15.0 1575.600000 77.379215 1450.0 1527.5 1618.0 1624.0 1636.0
    entropic 86400 15.0 1488.533333 6.599423 1466.0 1488.5 1490.0 1492.0 1493.0
    libfuzzer 86400 15.0 1498.333333 52.044852 1467.0 1485.5 1487.0 1488.0 1685.0
    aflfast 86400 15.0 1547.466667 93.821157 1466.0 1473.5 1479.0 1656.5 1674.0
    lafintel 86400 15.0 1495.466667 63.333434 1464.0 1470.5 1473.0 1474.5 1660.0
    fairfuzz 86400 15.0 1167.466667 165.574699 1091.0 1091.0 1091.0 1091.0 1616.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-parser summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 86400 15.0 11489.866667 41.560398 11378.0 11471.5 11503.0 11523.5 11534.0
    fastcgs_lm 86400 15.0 11326.733333 30.690777 11275.0 11308.5 11331.0 11351.0 11366.0
    symqemu 86400 15.0 11275.800000 35.246479 11216.0 11248.0 11272.0 11303.0 11330.0
    aflsmart 86400 15.0 11241.333333 28.629821 11202.0 11218.5 11242.0 11263.0 11296.0
    afl 86400 15.0 11227.666667 31.192185 11184.0 11212.0 11219.0 11240.0 11296.0
    mopt 86400 15.0 11211.200000 22.932198 11171.0 11195.0 11205.0 11228.5 11247.0
    aflfast 86400 15.0 11115.266667 35.311808 11051.0 11087.0 11114.0 11148.5 11158.0
    aflplusplus 86400 15.0 11121.266667 32.857847 11078.0 11095.0 11114.0 11146.5 11182.0
    entropic 86400 15.0 11069.400000 57.925815 10934.0 11043.5 11064.0 11101.5 11164.0
    lafintel 86400 15.0 10926.866667 62.521844 10822.0 10891.0 10920.0 10949.5 11062.0
    fairfuzz 86400 15.0 10690.800000 328.095631 10293.0 10359.0 10798.0 10977.0 11151.0
    libfuzzer 86400 15.0 10228.000000 112.657509 10024.0 10190.5 10235.0 10273.0 10428.0
    eclipser 86400 15.0 9897.666667 20.920484 9863.0 9885.0 9893.0 9903.5 9945.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4-2017-08-14 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 86400 15.0 3250.533333 45.566069 3165.0 3219.0 3260.0 3283.0 3334.0
    entropic 86400 15.0 2349.466667 48.111280 2259.0 2323.5 2352.0 2381.0 2414.0
    libfuzzer 86400 15.0 2266.600000 49.033225 2195.0 2236.0 2257.0 2294.0 2352.0
    afl 86400 15.0 2140.800000 55.447788 2074.0 2097.5 2142.0 2163.0 2277.0
    symqemu 86400 15.0 2133.933333 43.150015 2077.0 2096.5 2134.0 2157.0 2229.0
    aflsmart 86400 15.0 2122.933333 49.934910 2068.0 2093.0 2108.0 2139.5 2244.0
    lafintel 86400 15.0 2106.933333 43.561888 2057.0 2076.0 2088.0 2130.0 2191.0
    aflplusplus 86400 15.0 2068.333333 30.398935 2014.0 2060.0 2067.0 2084.0 2135.0
    fairfuzz 86400 15.0 2028.266667 45.689115 1921.0 2027.5 2044.0 2058.0 2069.0
    aflfast 86400 15.0 2030.400000 39.440371 1939.0 2004.5 2025.0 2055.0 2096.0
    mopt 86400 15.0 1956.066667 94.720994 1761.0 1893.0 1976.0 2018.5 2096.0
    fastcgs_lm 86400 15.0 1952.200000 71.914831 1828.0 1922.0 1966.0 1986.0 2092.0
    eclipser 86400 15.0 141.533333 44.788179 81.0 92.5 154.0 162.0 217.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2-2014-12-09 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 86400 15.0 2281.933333 12.919900 2247.0 2285.5 2287.0 2288.5 2293.0
    entropic 86400 15.0 2280.066667 8.396144 2271.0 2275.0 2278.0 2279.5 2302.0
    honggfuzz 86400 15.0 2274.466667 7.954035 2264.0 2269.0 2272.0 2277.0 2292.0
    symqemu 86400 15.0 2261.466667 7.605136 2251.0 2254.0 2262.0 2266.5 2274.0
    aflsmart 86400 15.0 2248.866667 4.778922 2240.0 2246.0 2249.0 2251.0 2261.0
    afl 86400 15.0 2247.733333 3.081434 2242.0 2245.5 2248.0 2249.5 2253.0
    fairfuzz 86400 15.0 2247.333333 2.742956 2243.0 2245.0 2247.0 2249.5 2251.0
    aflfast 86400 15.0 2246.200000 2.336053 2241.0 2245.0 2246.0 2247.0 2251.0
    aflplusplus 86400 15.0 2242.133333 4.340287 2234.0 2239.5 2241.0 2245.5 2249.0
    fastcgs_lm 86400 15.0 2231.666667 4.186145 2226.0 2228.0 2231.0 2235.0 2239.0
    lafintel 86400 15.0 2230.066667 11.099979 2212.0 2222.0 2230.0 2237.5 2248.0
    mopt 86400 15.0 2229.200000 3.233751 2225.0 2227.0 2228.0 2230.0 2237.0
    eclipser 86400 15.0 1933.200000 13.623509 1903.0 1924.0 1935.0 1944.0 1952.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fastcgs_lm 86400 15.0 17674.000000 218.534731 17339.0 17546.00 17690.0 17780.00 18169.0
    afl 86400 15.0 17241.066667 166.760507 16905.0 17169.00 17305.0 17328.50 17488.0
    mopt 86400 15.0 17218.266667 217.822164 16722.0 17167.50 17279.0 17320.00 17493.0
    aflsmart 86400 15.0 17228.600000 269.102158 16850.0 17021.00 17210.0 17357.50 17761.0
    symqemu 86400 15.0 16737.466667 239.949955 16283.0 16615.00 16730.0 16855.50 17246.0
    aflfast 86400 15.0 16625.800000 216.406100 16251.0 16471.50 16653.0 16786.00 16919.0
    aflplusplus 86400 15.0 16617.266667 284.190949 16176.0 16434.50 16577.0 16812.00 17164.0
    honggfuzz 86400 15.0 12636.533333 378.348076 11765.0 12395.00 12687.0 12871.00 13193.0
    lafintel 86400 15.0 12121.933333 537.601348 11332.0 11766.00 11884.0 12539.50 13260.0
    entropic 86400 15.0 11266.400000 679.365649 10181.0 10656.00 11162.0 11835.00 12461.0
    fairfuzz 86400 15.0 10771.533333 2226.481012 8573.0 8802.00 9391.0 12409.50 14752.0
    libfuzzer 86400 15.0 8824.333333 350.857860 8240.0 8547.00 8802.0 9017.00 9512.0
    eclipser 86400 14.0 5950.857143 781.753931 5400.0 5400.25 5403.5 6723.75 7218.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis-2017-12-11 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fastcgs_lm 86400 15.0 1016.466667 8.157964 1006.0 1011.0 1016.0 1018.0 1034.0
    aflsmart 86400 15.0 1016.600000 9.186947 1003.0 1011.0 1015.0 1018.0 1039.0
    mopt 86400 15.0 1016.466667 5.449334 1011.0 1013.0 1015.0 1017.0 1031.0
    symqemu 86400 15.0 1011.200000 8.937881 1002.0 1005.0 1009.0 1014.0 1033.0
    afl 86400 15.0 1010.933333 8.932018 1004.0 1005.5 1007.0 1012.5 1032.0
    aflplusplus 86400 15.0 1009.866667 8.492574 1001.0 1004.0 1006.0 1017.0 1026.0
    entropic 86400 15.0 979.533333 74.620819 795.0 998.5 1005.0 1011.0 1024.0
    fairfuzz 86400 15.0 1000.333333 12.965155 966.0 1000.5 1005.0 1007.5 1010.0
    honggfuzz 86400 15.0 1007.933333 16.307170 991.0 1000.0 1003.0 1006.0 1046.0
    aflfast 86400 15.0 1005.466667 8.798268 998.0 1000.0 1001.0 1009.0 1024.0
    lafintel 86400 15.0 984.133333 6.104643 973.0 979.0 984.0 987.5 994.0
    eclipser 86400 15.0 883.400000 37.339753 791.0 871.0 877.0 907.5 939.0
    libfuzzer 86400 15.0 886.533333 106.487334 790.0 790.0 792.0 994.0 1017.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2-2016-05-06 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    lafintel 86400 15.0 1094.066667 85.091939 965.0 1007.5 1122.0 1164.5 1189.0
    honggfuzz 86400 15.0 1098.200000 41.116906 1023.0 1083.0 1090.0 1112.5 1168.0
    symqemu 86400 15.0 1083.666667 28.336414 1032.0 1063.5 1089.0 1103.5 1133.0
    fastcgs_lm 86400 15.0 1084.266667 56.200491 1001.0 1071.0 1075.0 1089.5 1207.0
    afl 86400 15.0 1050.133333 36.392438 990.0 1021.5 1065.0 1073.5 1112.0
    aflsmart 86400 15.0 1034.733333 24.028158 994.0 1022.5 1030.0 1041.5 1085.0
    entropic 86400 15.0 1028.000000 53.088874 919.0 1004.5 1026.0 1047.0 1158.0
    mopt 86400 15.0 1036.733333 36.078222 1005.0 1013.5 1021.0 1053.0 1135.0
    aflplusplus 86400 15.0 1013.800000 20.805906 988.0 1002.0 1010.0 1018.5 1059.0
    libfuzzer 86400 15.0 1000.600000 25.751006 956.0 983.0 1000.0 1014.0 1054.0
    aflfast 86400 15.0 966.333333 17.520057 955.0 958.0 960.0 963.5 1013.0
    fairfuzz 86400 15.0 944.866667 21.111495 896.0 938.5 945.0 948.5 1004.0
    eclipser 86400 15.0 798.066667 34.691429 685.0 800.0 804.0 813.0 834.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 86400 15.0 335.333333 2.663689 330.0 333.5 334.0 338.0 339.0
    libfuzzer 86400 15.0 334.266667 2.282438 331.0 333.0 334.0 336.0 338.0
    aflsmart 86400 15.0 332.000000 2.171241 328.0 330.0 333.0 334.0 334.0
    fairfuzz 86400 15.0 330.933333 2.711527 324.0 329.5 331.0 333.0 335.0
    honggfuzz 86400 15.0 331.933333 2.186539 329.0 330.0 331.0 333.5 335.0
    afl 86400 15.0 330.400000 2.720294 325.0 329.5 330.0 331.5 336.0
    fastcgs_lm 86400 15.0 330.333333 2.870208 324.0 328.5 330.0 332.5 334.0
    symqemu 86400 15.0 331.066667 2.218966 329.0 329.0 330.0 333.0 334.0
    aflplusplus 86400 15.0 329.733333 3.195235 324.0 328.0 329.0 332.5 335.0
    lafintel 86400 15.0 329.733333 2.282438 326.0 329.0 329.0 331.5 335.0
    mopt 86400 15.0 330.000000 2.449490 325.0 329.0 329.0 331.5 335.0
    aflfast 86400 15.0 325.400000 3.202677 317.0 323.5 326.0 327.5 329.0
    eclipser 86400 15.0 314.666667 4.203173 308.0 312.0 313.0 318.5 321.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.