Psychosis Prognosis Predictor: A Continuous and Uncertainty-Aware Prediction of Treatment Outcome in First-Episode Psychosis

Importance Presently, clinicians face challenges in accurately predicting the prognosis of patients with psychosis. Although machine learning models have shown promising potential in individual-level outcome prediction, their practical implementation as tools for real-world clinical practice has been hindered by several limitations. These limitations include difficulties in predicting multiple clinical outcomes, effectively capturing the evolving status of patients over time, and establishing trust in machine predictions. Addressing these shortcomings is crucial for responsibly leveraging machine learning in clinical decision-making for psychosis prognosis. Objective We propose and evaluate a multi-task recurrent neural network architecture specifically designed to overcome these challenges, enabling trustworthy prediction of psychosis prognosis. By developing a model that can effectively handle multiple clinical outcomes, capture the dynamic nature of patients’ status over time, and instill confidence in its predictions, we aim to provide clinicians with a robust tool for making informed and responsible decisions in the treatment of psychosis. Design, setting, and participants The study sample comprised 446 individuals, aged between 18 and 40 years, diagnosed with first-episode psychosis and participating in the OPTiMiSE study. To predict the likelihood of remission, we selected a range of multimodal baseline variables. These variables included PANSS scores (Positive and Negative Syndrome Scale), diagnostic subclass, duration of untreated psychosis, age at onset, sex, as well as physical health and lifestyle variables. The treatment outcome measures assessed at both week 4 and week 10 encompassed symptomatic remission, clinical global remission, and functional remission. Main outcome and measures we devised a recurrent neural network architecture incorporating long short-term memory (LSTM) units to facilitate outcome prediction by leveraging multimodal baseline variables and clinical data collected at multiple time points. To account for model uncertainty, we employed a novel fuzzy logic approach to integrate the level of uncertainty into individual predictions. To assess the predictive performance of our model, we employed rigorous evaluation techniques, including repeated 10-fold cross-validation and leave-one-site-out cross-validation. Results When solely utilizing pre-treatment patient status to forecast various outcomes in 4 weeks after starting the treatment, the leave-one-site-out validation process yielded area under the receiver operating characteristic curve (AUC) values ranging from 0.62 to 0.67. Notably, the predictive performance witnessed ∼0.04 improvement upon the inclusion of 1-week follow-up patient status, resulting in AUC values ranging from 0.66 to 0.72. Regarding the prediction of outcomes 10 weeks after the start of treatment, the models constructed solely with pre-treatment patient status achieved AUC values between 0.56 and 0.64. However, by incorporating follow-up patient status (specifically after 1, 4, and 6 weeks), the performance of the models was enhanced, resulting in AUC values of 0.72 to 0.74. Furthermore, after incorporating prediction uncertainties and stratifying the model decisions based on model confidence, we could achieve accuracies above 0.8 for ∼50% of patients in five out of six clinical scenarios. Conclusions and Relevance Our approach involved constructing prediction models utilizing a flexible neural network architecture tailored to clinical scenarios derived from a time series dataset. One crucial aspect we incorporated was the consideration of uncertainty in individual predictions, which enhances the reliability of decision-making based on the model’s output. Through our study, we provided compelling evidence showcasing the significance of leveraging time series data for achieving more accurate treatment outcome prediction in the field of psychiatry. Therefore, we advocate for the inclusion of time series data in future individualized prediction modeling endeavors within psychiatric research, as it holds substantial promise in improving prognostic accuracy and informing personalized treatment approaches. ### Competing Interest Statement RSK reports consulting fees from Alkermes, Sunovion, Gedeon-Richter, and Otsuka. ### Funding Statement This work was supported by ZonMw (project ID 63631 0011) and by a research grant from the AI for Health working group of the TU/e-WUR-UU-UMCU (EWUU) alliance. ### Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: OPTiMiSE dataset: I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes All data produced in the present study are available upon reasonable request to the authors.