The Heads Up Checkup Digital Mental Health & Behavioral Risk Screening System: Clinical Inter-Rater Reliability for Identifying Youth in Crisis

ABSTRACT The present study evaluated the inter-rater reliability of the Heads Up Checkup (HCU), a brief digital mental health and behavioral adaptive screening system designed for use in primary care and diverse school settings. Two independent licensed clinical psychologists reviewed a random sample of thirty (N=30) HCU clinical screening reports (CSRs) of 13-14 year old adolescents drawn from a larger sample (N=846) enrolled in a predominantly Hispanic public middle school in California. Results showed strong inter-rater agreement (Fleiss kappa=.93) between clinician ratings and the screener’s priority risk index (HPI) in identifying students “in crisis.” In addition, clinicians’ ratings of confidence in their priority judgments were found to be significantly higher for the high-risk “in crisis” cases. Reasonable evidence of convergent validity emerged due to the significant correlations between clinician ratings of psychological distress and the HPI. Overall findings suggest that as an online universal school-based screener, the HCU has valid utility for identifying high-risk “in crisis” cases which can translate into timely and pragmatic real-world therapeutic solutions for diverse populations of adolescents. Future research directions with respect to the refinement of measurement issues and feasibility of implementation of population-level online screeners in schools are discussed. Keywords: Adolescent psychological distress, youth mental health risk assessment, inter-rater reliability, convergent validity.