Complementing Datasets for Recognition and Analysis of Affect in Speech

The paper presents a framework for an automatic system that infers affective states from their non-verbal expressions in speech. The goal was to infer affective states occurring in real scenarios, i.e. affective states which are common in everyday lives and in human-computer interactions, can occur simultaneously and whose level of expression can change dynamically and asynchronously over time, as well as investigating the generalization to a very large variety of affective states and to other languages. The framework was based on two complementing datasets, Mind Reading and Doors, which were used for the design and validation of the system. The chosen datasets provided data in two languages, by speakers of both genders and of all age groups, actors and non-actors. The data comprised of acted and naturally evoked affective states, with varied text and text repetitions, labeled affective states and multi-modal information, single sentences and sustained interactions, a large variety of affective states and nuances of subtle affective states, in two different languages (English and Hebrew). The paper shortly describes the datasets, their advantages and disadvantages, and how their combination was used in order to achieve the design goals.