A Quality Framework for Statistical Algorithms

Executive summary As national statistical offices (NSOs) modernize, interest in integrating machine learning (ML) into official statisticians’ toolbox is growing. In the 2018 Blue Skies Thinking Network paper (UNECE 2018), two issues were identified: the potential loss of transparency from the use of “black-boxes” and the need to develop a quality framework. The goal of Work Package 2 of the High-Level Group for the Modernisation of Official Statistics’ ML project was to address these two challenges. Many quality frameworks exist; however, they were conceived with traditional methods in mind, and they tend to target statistical outputs. Currently, ML methods are being looked at for use in processes producing intermediate outputs, which lead to a final statistical output. These processes include, for example, coding, imputation and editing, which do not directly produce statistical outputs, but whose results are used downstream to ultimately produce final outputs.