Using large language models (LLMs) to apply analytic rubrics to score post-encounter notes.

Area of Science:

• Artificial Intelligence in Medical Education
• Natural Language Processing for Clinical Assessment

Background:

• Large language models (LLMs) show potential for applications in medical education.
• Objective Structured Clinical Examinations (OSCEs) are a key component of medical training.

Purpose of the Study:

• To evaluate the ability of LLMs to score post-encounter notes (PNs) using an analytic rubric.
• To refine methods for accurate and consistent LLM-based scoring of clinical notes.

Main Methods:

• Seven LLMs scored five PNs with varying performance levels.
• Iterative experimental design tested different prompting strategies and temperature settings.
• LLM scores were compared against expected rubric-based results.

Main Results:

• Consistent scoring required multiple rounds of prompt refinement and structured approaches.
• Low-temperature settings improved score variability.
• LLMs occasionally required external calculation for total scores, but the final approach achieved consistent accuracy.

Conclusions:

• LLMs can reliably apply analytic rubrics to PNs with careful prompt engineering.
• LLMs show potential as scalable, automated scoring tools in medical education.
• Further research is needed for LLM application with holistic rubrics.