Automatic diagnosis of multi-task in essential tremor: Dynamic handwriting analysis using multi-modal fusion neural network

Essential tremor (ET) is one of the most common movement disorders, and patients with ET have more than a fourfold increased risk of developing Parkinson's disease (PD). Currently, the handwriting assessment remains the gold standard for ET diagnosis and quantification of tremor severity. However, the traditional evaluation of tremors is subjective and partial, primarily limited by the experience of the neurologist and the progression of the disease course. With advanced sensing and computer technology development, digital pad-based electronic handwriting analysis can provide clinical information on pen tip kinematics. This study proposes a novel spatial-temporal-spectral fusion neural network (STSNet) for multi-task fine-grained assessment of tremor severity. The model can efficiently fuse complementary information from static handwriting images and dynamic multi-sensory fusion signals. Specifically, we design a static spatial branch for learning the global spatial features of the electronic handwriting image and a dynamic temporal branch to learn the spectrogram features in the time- frequency domain. The spectrograms are obtained by the transformation algorithm of the kinematic signals of the nib sensors to represent the spectral energy associated with tremor. In addition, we also validate the utility of the attention mechanism, transfer learning strategy, and the introduction of demographic features-based prior knowledge by ablation studies. STSNet obtains an average accuracy of 97.33-97.39% (five categories) on our collected database containing 147 ET patients. In addition, we test the robustness and effectiveness of the proposed method in three publicly available databases of PD diagnostics. These results show that STSNet outperforms other advanced works, which can automatically provide objective and accurate scores for ET symptom assessment or PD diagnosing.(c) 2023 Elsevier B.V. All rights reserved.