A Novel Evaluation Model for Assessing ChatGPT on Otolaryngology-Head and Neck Surgery Certification Examinations: Performance Study

Cai Long, Kayle Lowe,Jessica Zhang, Andre dos Santos, Alaa Alanazi, Daniel O'Brien,Erin Wright,David Cote

JMIR MEDICAL EDUCATION(2024)

引用 0|浏览0
暂无评分
摘要
Background: ChatGPT is among the most popular large language models (LLMs), exhibiting proficiency in various standardized tests, including multiple-choice medical board examinations. However, its performance on otolaryngology-head and neck surgery (OHNS) certification examinations and open-ended medical board certification examinations has not been reported. Objective: We aimed to evaluate the performance of ChatGPT on OHNS board examinations and propose a novel method to assess an AI model's performance on open-ended medical board examination questions. Methods: Twenty-one open-ended questions were adopted from the Royal College of Physicians and Surgeons of Canada's sample examination to query ChatGPT on April 11, 2023, with and without prompts. A new model, named Concordance, Validity, Safety, Competency (CVSC), was developed to evaluate its performance. Results: In an open-ended question assessment, ChatGPT achieved a passing mark (an average of 75% across 3 trials) in the attempts and demonstrated higher accuracy with prompts. The model demonstrated high concordance (92.06%) and satisfactory validity. While demonstrating considerable consistency in regenerating answers, it often provided only partially correct responses. Notably, concerning features such as hallucinations and self-conflicting answers were observed. Conclusions: ChatGPT achieved a passing score in the sample examination and demonstrated the potential to pass the OHNS certification examination of the Royal College of Physicians and Surgeons of Canada. Some concerns remain due to its hallucinations, which could pose risks to patient safety. Further adjustments are necessary to yield safer and more accurate answers for clinical implementation.
更多
查看译文
关键词
medical licensing,otolaryngology,otology,laryngology,ear,nose,throat,ENT,surgery,surgical,exam,exams,response,responses,answer,answers,chatbot,chatbots,examination,examinations,medical education,otolaryngology/head and neck surgery,OHNS,artificial intelligence,AI,ChatGPT,medical examination,large language models,language model,LLM,LLMs,wide range information,patient safety,clinical implementation,safety,machine learning,NLP,natural language processing
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要