230 Inter-rater reliability assessment of OPTI-3S (criteria for optimising medicines by stopping, stepping down, or switching to safer alternatives)

Abstract Background The OPTI-3S criteria comprise newly developed consensus-based statements identifying potentially inappropriate medications for hospitalised frail older adults. This study assessed the inter-rater reliability of the OPTI-3S core list. Methods Data were collated from ten patient cases in the publicly archived ‘The Irish Longitudinal Study on Ageing’ (TILDA) dataset (Wave 3). Five healthcare professionals assessed medication appropriateness across the cases according to the OPTI-3S statements, indicating the applicable statement(s). Their assessments were compared against a ‘Gold Standard (GS) assessment’ consisting of a combined assessment of a research team member and a clinically experienced academic researcher, who independently assessed the medications’ appropriateness. Concordance between the GS rating and other raters was assessed by kappa statistics. Major discrepancies were identified when a rater applied a statement >2 times more or less frequently than GS [1]. Results The ten patients’ median age was 74 (Interquartile Range: 70–80) years, and 40% were female. The median (IQR) number of medications was 10 (9–12). Cohen’s kappa coefficient ranged from 0.65 to 0.86, indicating substantial to perfect interrater reliability when agreement was measured between the GS rating and any other rater. The overall mean (95% CI) Fleiss kappa coefficient between all raters was 0.67 (0.65,0.69), indicating substantial interrater reliability. The mean Fleiss kappa coefficient (95% CI) between the GS rating and geriatricians was 0.63 (0.60, 0.67). A good Fleiss kappa coefficient of 0.81 (0.75, 0.86) was found between the GS rating and pharmacists. Major discrepancies in applying seven statements were noted between the GS rating and three raters. Three of these statements were misinterpreted, indicating revised wording is needed. Conclusion The OPTI-3S core list shows substantial inter-rater reliability between experts with no previous experience using it. However, further testing with a larger number of clinical cases and raters is required.