An Enhanced Dataset for Inferential Emotion Tracking in Humans and Machines

Emotion perception is fundamental to human experience, and it reveals both the sophistication and richness of vision. Although emotion perception depends in part on face processing, it also hinges critically on background scene context (e.g., Chen & Whitney, 2019, 2020, 2021). However, how the visual system uses background scene context to modulate and assign perceived emotion is poorly understood. AI models have the potential to be uniquely useful in this regard: they could shed light on how the human brain processes emotion. However, AI and machine learning approaches fail to incorporate or capture context information. There are not even any AI training datasets that focus on context information in emotion processing. Moreover, insufficient annotators are recruited in existing studies, introducing likely human biases in the annotations. Previous datasets are therefore severely limited. For AI to be useful in studying human perception of emotion, AI needs extensive psychophysical data for training, including data about the background context. Here, our goal was to define and create the kind of psychophysical data that AI models can use for training. In this study, we created a novel emotion tracking dataset with more than 100 video clips, which provide character and contextual information with ratings of both continuous arousal/valence and discrete emotion categories on a frame-by-frame basis. A large number of annotators were recruited to avoid idiosyncratic biases. Our psychophysically controlled stimuli improve the psychological validity of the data used in AI research, improve the statistical power, and reduce bias and artifacts that accompanied previous data sets used in machine learning. Further, we also propose a baseline AI approach, using a transformer model, which achieves state-of-the-art performance on emotion recognition tasks. This model could be useful in helping guide an understanding of how humans use contextual information to perceive emotion in natural scenes.