Prompt Engineering GPT-4 to Answer Patient Inquiries: A Real-Time Implementation in the Electronic Health Record across Provider Clinics

Background The integration of Large Language Models like GPT-4 in healthcare has opened new avenues for improving patient-provider communication. However, the effectiveness of prompt engineering strategies, which have been shown to affect the quality and accuracy of model output, remains unexplored in this context. This study aims to evaluate the impact of manual versus semiautomated prompt engineering to improve the usability of AI-generated responses to patient inquiries with real-time integration into an electronic health record. Methods A pre-post study over eight months was conducted at University of Wisconsin Health, involving 27 providers across multiple specialties. The study compared GPT-4 in a pre-period use of manual prompts with a post-period semi-automated engineered prompt that incorporated iterative design and prompt evaluation scoring. Testing by informaticists was completed before deployment of the new prompt. The primary outcome was the number of AI-generated draft messages used in a mixed effects model accounting for multiple messages by same provider. Secondary outcomes included message editing metrics and sentiment analysis. Results Of the 7,605 draft messages generated by GPT-4 and seen by providers during the study period, 17.5% (n=1,327) were used by the providers and 2.6% (n=202) were left identical or nearly identical in editing by the providers. The number of messages used decreased in the post-period with the new prompt (beta coefficient -0.10; 95% CI: -0.11 - -0.09, p<0.01); however, there was a reduction in negative sentiment with an odds ratio of 0.43 (95% CI: 0.36 - 0.52, p<0.01) with the new prompt design. Discussion The decrease in negative sentiment with the new prompt demonstrated improvement in the AI-generated content quality but usage remained low by providers. This study highlights a need for better alignment between prompt engineering and human factors engineering. Trial Registration number Not Applicable ### Competing Interest Statement The authors have declared no competing interest. ### Funding Statement The study was funded by the NIH National Center for Advancing Translational Sciences (UL1TR002373) ### Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: The Institutional Review Board at the University of Wisconsin-Madison and provided approval with an exemption status. I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes The original data derives from the institutions electronic health record (EHR) and contains patients protected health information (PHI). Data are available from the University of Wisconsin Health for researchers who meet the criteria for access to confidential data and have a data usage agreement and IRB approval with the health system. Only the prompt engineering software with the library of prompts is open-source and available at .