Performance Analysis with Bayesian Inference.

Statistics are part of any empirical science, and performance analysis is no exception. However, for non-statisticians, picking the right statistical tool to answer a research question can be challenging; each statistical tool comes with a set of assumptions, and it is not clear to researchers what happens when those assumptions are violated. Bayesian statistics offers a framework with more flexibility and with explicit assumptions. In this paper, we present a method to analyse benchmark results using Bayesian inference. We demonstrate how to perform a Bayesian analysis of variance (ANOVA) to estimate what factors matter most for performance, and describe how to investigate what factors affect the impact of optimizations. We find the Bayesian model more flexible, and the Bayesian ANOVA's output easier to interpret.