Wearable Multimodal - Serious Game System For Hand and Cognitive Rehabilitation After Stroke

Abstract Objective: Stroke often leads to both motor control and cognitive dysfunction, and effective rehabilitation requires keeping patients engaged and motivated. We introduce a wearable multimodal system based on force myography, electromyography, and inertial sensing with two associated serious games for stroke rehabilitation of twelve hand movements related to activities of daily living and the Fugl Meyer Assessment.Methods: In the ‘Find the Sheep’ serious game, patients performed corresponding hand movements to select the correct sheep card, and in the ‘Best Salesman’ serious game, patients performed corresponding hand movements to grab specific food and drink items in a store. A multi-sensor fusion model was developed for movement classification via linear discriminant analysis. Ten stroke patients with mild to moderate motor impairments (Brunnstrom Stage for Hand II-VI) performed validation testing, and effectiveness was evaluated by movement classification accuracy and qualitative patient questionnaires.Results: Classification accuracy for twelve movements using combined force myography, electromyography, and inertial sensing was 81.0%, and accuracies for using electromyography, force myography, or inertial sensing alone were 69.6%, 63.2%, and 47.8%, respectively. All patients reported that they were more enthusiastic about rehabilitation while playing serious games than conventional rehabilitation, and a majority reported the wearable multimodal-based system was easier to wear than a sensorized data glove. Significance: Results showed that multi-sensor fusion could improve hand gesture classification accuracy for stroke patients and demonstrated that the proposed wearable multimodal-serious game system could potentially facilitate upper extremity rehabilitation and cognitive training after stroke.