SELFI: Evaluation of Techniques to Reduce Self-report Fatigue by Using Facial Expression of Emotion.

This paper presents the SELFI framework which uses information from a range of indirect measures to reduce the burden on users of context-sensitive apps in the need to self-report their mental state. In this framework, we implement multiple combinations of facial emotion recognition tools (Amazon Rekognition, Google Vision, Microsoft Face), and feature reduction approaches to demonstrate the versatility of the framework in facial expression based emotion estimation. The evaluation of the framework involving 20 participants in a 28-week in-the-wild study reveals that the proposed framework can estimate emotion accurately using facial image ( $$83\%$$ and $$81\%$$ macro-F1 for valence and arousal, respectively), with an average reduction of $$10\%$$ self-report burden. Moreover, we propose a solution to detect the performance drop of the model developed by SELFI, during runtime without the use of ground truth emotion, and we achieve accuracy improvements of 14%.