A Siren Song of Open Source Reproducibility, Examples from Machine Learning.

As reproducibility becomes a greater concern, conferences have largely converged to a strategy of asking reviewers to indicate whether code was attached to a submission. This represents a broader pattern of implementing actions based on presumed ideals, without studying whether those actions will produce positive results. We argue that focusing on code as a means of reproduction is misguided if we want to improve the state of reproducible and replicable research. In this study, we find this focus on code may be harmful - we should not force code to be submitted. Furthermore, there is a lack of evidence that conferences take effective actions to encourage and reward reproducibility. We argue that venues must take more action to advance reproducible machine learning research today.