Mudra: A Multi-Modal Smartwatch Interactive System with Hand Gesture Recognition and User Identification
IEEE INFOCOM 2022 - IEEE Conference on Computer Communications(2022)
摘要
The great popularity of smartwatches leads to a growing demand for smarter interactive systems. Hand gesture is suitable for interaction due to its unique features. However, the existing single-modal gesture interactive systems have different biases in diverse scenarios, which makes it intractable to be applied in real life. In this paper, we propose a multi-modal smartwatch interactive system named Mudra, which fuses vision and Inertial Measurement Unit (IMU) signals to recognize and identify hand gestures for convenient and robust interaction. We carefully design a parallel attention multi-task model for different modals, and fuse classification results at the decision level with an adaptive weight adjustment algorithm. We implement a prototype of Mudra and collect data from 25 volunteers to evaluate its effectiveness. Extensive experiments demonstrate that Mudra can achieve 95.4% and 92.3% F1-scores on recognition and identification tasks, respectively. Meanwhile, Mudra can maintain stability and robustness under different experimental settings.
更多查看译文
关键词
Mudra,multimodal smartwatch interactive system,hand gesture recognition,user identification,smarter interactive systems,existing single-modal gesture,convenient interaction,robust interaction,parallel attention multitask model,different modals
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要