Mental models of AI-based systems: User predictions and explanations of image classification results

Humans should be able work more effectively with artificial intelligence-based systems when they can predict likely failures and form useful mental models of how the systems work. We conducted a study of human’s mental models of artificial intelligence systems focusing on a high-performing image classifier. Participants viewed individual labeled images in one of three general classes and then tried to predict whether the classifier would label it correctly. Participants were able to begin performing this task at levels much better than chance, 69% correct. However, after 137 trials with feedback, their performance improved a small, but statistically significant amount to 73%. Analysis of these results and comments indicated that humans were using their own perceptions of the images as first-approximation proxies. ‘Projecting’ human characteristics onto a computer might be considered a cognitive bias, but in this task, the strategy seemed to yield good initial results. This might be called effective anthropomorphism. Participants sometimes used this strategy both implicitly and explicitly. The paper includes discussion of why this strategy might have worked better than alternatives, why further learning was quite difficult, and what assumptions about similarities between human perception and image classification systems may in fact be correct.