Large Language Model-Based Assessment of Clinical Reasoning Documentation in the Electronic Health Record Across Two...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Reasoning Assessment

Background:

• Clinical reasoning (CR) is a critical physician skill, yet feedback on its documentation in electronic health records (EHRs) is often limited.
• Artificial intelligence (AI) presents a promising solution to bridge this feedback gap in medical education and practice.

Purpose of the Study:

• To develop and evaluate AI-driven assessments for clinical reasoning (CR) documentation within EHRs.
• To compare the performance of Named Entity Recognition (NER), logic-based models, and Large Language Models (LLMs) across two institutions.

Main Methods:

• Utilized a retrospective and prospective corpus of internal medicine resident admission notes from NYU and UC.
• Developed and compared AI models including NER, a logic-based model, and two LLMs (NYUTron and GatorTron) for CR assessment.
• Assessed model performance using F1-scores, AUROC, and AUPRC on differential diagnosis and explanation of reasoning domains.

Main Results:

• At NYU, the NYUTron LLM demonstrated strong performance in assessing differential diagnosis (D0, D2) and explanation of reasoning (EA2).
• At UC, the NER-based logic model excelled in differential diagnosis assessment (D0, D1, D2), while GatorTron LLM performed best for EA2.
• Model performance varied across tasks and institutions, with some requiring adaptive strategies like stepwise approaches or binary classification.

Conclusions:

• This multi-institutional study is the first to apply LLMs for assessing EHR clinical reasoning documentation.
• Developed AI tools show potential for enhancing physician feedback on clinical reasoning, supporting medical education and practice.
• Lessons learned indicate the generalizability of these AI approaches across different institutional settings.