A NLP-based semi-automatic identification system for delays in follow-up examinations: an Italian case study on clinical referrals

This study aims to propose a semi-automatic method for monitoring the waiting times of follow-up examinations within the National Health System (NHS) in Italy, which is currently not possible to due the absence of the necessary structured information in the official databases. A Natural Language Processing (NLP) based pipeline has been developed to extract the waiting time information from the text of referrals for follow-up examinations in the Lombardy Region. A manually annotated dataset of 10 000 referrals has been used to develop the pipeline and another manually annotated dataset of 10 000 referrals has been used to test its performance. Subsequently, the pipeline has been used to analyze all 12 million referrals prescribed in 2021 and performed by May 2022 in the Lombardy Region. The NLP-based pipeline exhibited high precision (0.999) and recall (0.973) in identifying waiting time information from referrals’ texts, with high accuracy in normalization (0.948-0.998). The overall reporting of timing indications in referrals’ texts for follow-up examinations was low (2