Using Natural Language Processing to Identify Key Values in Internal Medicine-Pediatrics Residency Applications.

Purpose: Recruitment of excellent medical residents is a priority for all residency programs, but studies suggest traditional quantitative evaluation poorly predicts resident performance. Furthermore, current metrics have been found to be socioeconomically and racially biased. Several organizations including the Association of American Medical Colleges (AAMC) have urged adoption of holistic review to consider the whole applicant in relation to a specific residency’s goals and values. While holistic review has been used successfully in some residency programs to decrease bias in the selection process, it is limited by its time-consuming nature and has yet to be widely adopted. Approach: Previously, we used a modified Delphi method to identify and internally validate 11 values of successful residents for our internal medicine-pediatrics residency program (MP) in the Mountain West region: academic strength; intellectual curiosity; compassion; communication; work ethic; teamwork; leadership; self-awareness; diversity, equity, and inclusion (DEI); professionalism; and adaptability.1 Using residency applications to our program from 2015 to 2021, we created a program to extract relevant snippets of text pertaining to these values. These snippets were then annotated by expert reviewers and assigned to specific values. Using natural language processing (NLP), these annotated snippets were used to train a machine learning algorithm (MLA) to classify snippets into values. We then tested the MLA by comparing a set of snippets annotated by humans to those generated by the MLA. We also analyzed snippets that were initially dismissed by the human annotator but were annotated by the MLA and vice versa to improve concordance. Outcomes: Overall, net agreement between human and MLA was 59.1% (n = 7,440) with range of 0%–78.5% when analyzed per value. Four values achieved agreement, defined as > 66% overlap: academic strength (78.5%; n = 522); leadership (68.9%; n = 694); communication (68.4%; n = 1,350); and justice, equity, diversity, and inclusion (JEDI) (67.8%; n = 769). Two values achieved agreement between 50% and 70% (work ethic: 65.6%, n = 1,048; and compassion: 53.2%, n = 690). Three values were poorly captured with agreement below 50% (flexibility: 40.4%, n = 532; teamwork: 40.4%, n = 354; and professionalism: 15.4%, n = 225). The remaining 3 values of self-awareness, intellectual curiosity, and humility (n = 376 total) were not annotated by the MLA due to insufficient pattern recognition. Six hundred and five snippets in the training set were annotated with a value by the MLA but discarded by the human annotator. These snippets were manually reviewed a second time. Forty-five percent of these annotations were deemed to be true positives for the MLA and false negatives by the human. When adjusting for the originally missed annotations, overall agreement increased to 62.8% and work ethic rose to the level of agreement (67.7%). Significance: We created an MLA that is able to identify values that have been validated as important for residency success. We will continue to refine the MLA to increase the agreement of the remaining values. Once the MLA is able to reliably identify these values, we plan to generate a screening tool based on these values that could yield a quantitative score for our program to use in an initial applicant screen for interview selection. This proof-of-concept work shows that NLP has potential to be an effective tool for residency application and selection. Such a tool would create a means to implement holistic review on a broad level and transform application screening, replacing current-biased evaluation measures.