High-level Features for Multimodal Deception Detection in Videos

Deception (the action of deliberately cause someone to believe something that is not true) can have many different repercussions in daily life. However, deception detection is an inherently complex task for humans. Due to this, not only there is uncertainty on which features should be used as cues for automatic deception detection, but labeled data is scarce. In this paper, we explore typical features that can be extracted from videos for affective computing and study their performance for deception detection in videos. Additionally, we perform a study of different multimodal fusion methods meant to improve the results obtained by using the different sets of extracted features separately, including a novel set of methods based on boosting. For this study, high-level features are extracted with open automatic tools for the visual, acoustical and textual modalities, respectively. Experiments are conducted using a real-life trial dataset for deception detection, as well as a novel Mexican deception detection dataset using Spanish as the spoken language.