A/B Testing in the Small: An Empirical Exploration of Controlled Experimentation on Internal Tools.

Previous research on A/B testing and continuous experimentation has mostly focused on large-scale settings with company-external customers. However, work efficiency and satisfaction for company co-workers may be highly related to internal tools and their fit-for-use. In this study, we therefore explore A/B testing for online services exclusively used by company co-workers. We A/B tested two versions of Customer Admin, a tool that helps 34 500 IKEA co-workers to interact with customer data. The study comprised i) stakeholder interviews to understand objectives and phrase the experimentation goals, and ii) A/B test execution where data was collected and processed from approximately 350 users for 33 days. While the user base is relatively large for this internal system, the primary metric data collected was too scarce to allow distinction between the two versions. However, secondary metrics and a user questionnaire suggest that the users are more efficient in the new menu design and that the users prefer it to the old. We conclude that A/B testing requires lots of data, making it less feasible for internal users, also for large, global organizations. Thus, we propose combining quantitative and qualitative evaluation of internal tools and propose further research on how to adapt A/B testing for smaller-scale settings.