Tracking Performance of Graal on Public Benchmarks

For the past three years, we have used several public Java benchmarks (DaCapo, ScalaBench, Renaissance) to track the performance changes introduced by the daily development changes of the Graal compiler. The goal of the talk is to first outline and summarize the history of the performance changes themselves, and then open a discussion on the usefulness of such testing for the compiler development process, for example on questions such as (1) what changes are useful to report (or not), (2) what changes are missed by the public benchmarks, or (3) what aspects of the compiler behavior make such testing difficult (or easy).