Using Large Language Model Artificial Intelligence to Enhance Clinical Competency Committee Insight

Artificial intelligence, particularly large language models, has the potential to enhance the evaluation of residents, yet few studies have compared artificial intelligence to humans in the analysis of large volumes of clinical evaluations. The objective of this study was to evaluate the use of large language model (LLM) artificial intelligence (AI) on the evaluation of resident feedback.

The LLM Llama-3.1 70B was used on a secure cluster to conduct an analysis of emergency medicine (EM) educator feedback for 31 EM residents at a residency program in the Southeastern United States. The large language model generated assessments summarizing resident strengths, weaknesses, and milestone-based performance. Members of a clinical competency committee (CCC) blinded to the study hypothesis were surveyed to assess their perceptions on the quality, accuracy, specificity, and usefulness of the AI-generated content compared to the human-generated content.

Human-generated assessments averaged 79 words, while AI-generated assessments averaged 391 words. Seventy percent of the CCC completed the survey, rating the AI-generated content more favorably for quality, accuracy, and specificity compared to the human-generated content. Usefulness of the human-generated content was reported as good or very good by 71.4%, while usefulness of the AI-generated content was rated as acceptable (54.1%) or unfavorable (28.6%).

Artificial intelligence can generate EM resident assessments with comparable or superior ratings of quality, accuracy, and specificity relative to human-generated assessments. This highlights the potential of AI-driven evaluations to streamline educator review processes, reducing workload without sacrificing integrity.