Developing an artificial intelligence–based diagnostic model of headaches from a dataset of clinic patients' records

Abstract Objective We developed an artificial intelligence (AI)‐based headache diagnosis model using a large questionnaire database from a headache‐specializing clinic. Background Misdiagnosis of headache disorders is a serious issue and AI‐based headache diagnosis models are scarce. Methods We developed an AI‐based headache diagnosis model and conducted internal validation based on a retrospective investigation of 6058 patients (4240 training dataset for model development and 1818 test dataset for internal validation) diagnosed by a headache specialist. The ground truth was the diagnosis by the headache specialist. The diagnostic performance of the AI model was evaluated. Results The dataset included 4829/6058 (79.7%) patients with migraine, 834/6058 (13.8%) with tension‐type headache, 78/6058 (1.3%) with trigeminal autonomic cephalalgias, 38/6058 (0.6%) with other primary headache disorders, and 279/6058 (4.6%) with other headaches. The mean (standard deviation) age was 34.7 (14.5) years, and 3986/6058 (65.8%) were female. The model's micro‐average accuracy, sensitivity (recall), specificity, precision, and F ‐values for the test dataset were 93.7%, 84.2%, 84.2%, 96.1%, and 84.2%, respectively. The diagnostic performance for migraine was high, with a sensitivity of 88.8% and c ‐statistics of 0.89 (95% confidence interval 0.87–0.91). Conclusions Our AI model demonstrated high diagnostic performance for migraine. If secondary headaches can be ruled out, the model can be a powerful tool for diagnosing migraine; however, further data collection and external validation are required to strengthen the performance, ensure the generalizability in other outpatients, and demonstrate its utility in real‐world settings.