Recognizing Conversational State from the Eye Using Wearable Eyewear

During a naturalistic conversation, we can easily perceive whether a person is speaking or listening and when they are about to start or finish speaking, which we refer to as conversational states in this paper. Building this ability into a machine can be helpful in a wide variety of human-machine collaboration contexts for effective communication. A wealth of evidence from psychology and neuroscience suggests that the eye is a reliable window into human internal states since it senses the perceived changes in the outside world. In this study, we investigate the relationship between eye behavior and four conversational states for the first time and examine the viability of automatically recognizing the conversational states. The results demonstrate that eye center relative to the head is not a good indicator to distinguish conversational states, but the distribution, frequency and duration of eye states are strongly correlated with conversational states while eyelid is associated with conversational state in certain extent. The accuracy of recognition of the four conversational states using the proposed eye behaviors is well above chance level, and outperforms baselines using pupillary response, pupil center position, and blink rate. This finding suggests that eye state and eyelid shape contains valuable information about conversational states but have been overlooked in previous studies. It is promising to include eye behavior in wearable interfaces for dialogue systems and for social signal processing, with the further advantages of being 'always on', less sensitive to luminance variability, and better privacy preserving compared with using facial image and speech data.