A multi-modal imaging system for simultaneous measurement of speech articulator kinematics for bedside applications in clinical settings

A critical step toward a neurological understanding of speech generation is to relate neural activity to the movement of articulators. Here, we describe a noninvasive system for simultaneously tracking the movement of the lips, jaw, tongue, and larynx for human neuroscience research carried out at the bedside. We combined three methods previously used separately: videography to track the lips and jaw, electroglottography to monitor the larynx, and ultrasonography to track the tongue. To characterize this system, we recorded articulator positions and acoustics from six speakers during production of nine American English vowels. We describe processing methods for the extraction of kinematic parameters from the raw signals and methods to account for artifacts across recording conditions. To understand the relationship between kinematics and acoustics, we used regularized linear regression between the vocal tract kinematics and speech acoustics to identify which, and how many, kinematic features are required to explain both across vowel and within vowel acoustics. Furthermore, we used unsupervised matrix factorization to derive "prototypical" articulator shapes, and use them as a basis for articulator analysis. These results demonstrate a multi-modal system to non-invasively monitor speech articulators for clinical human neuroscience applications and introduce novel analytic methods for understanding articulator kinematics.