Development and Validation of a Machine Learning Model for Automated Assessment of Resident Clinical Reasoning...

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine
• Clinical Informatics

Background:

• Physicians receive limited feedback on clinical reasoning (CR) documentation quality.
• Machine learning (ML) and natural language processing (NLP) have been applied to standardized cases but not yet in clinical settings for CR assessment.

Purpose of the Study:

• To develop and validate a ML model for automated assessment of CR documentation quality in residents' admission notes within a clinical environment.
• To address the gap in providing regular feedback on CR documentation to residents.

Main Methods:

• A ML model was developed using 414 internal medicine resident admission notes, rated with a CR documentation rubric.
• Natural language processing (NLP) software (cTAKES) extracted disease entities, and human review generated CR terms.
• The model was validated on 9591 retrospective notes, with reliability assessed using Cohen's kappa.

Main Results:

• The logistic regression model achieved an area under the curve of 0.88, accuracy of 0.79, and Cohen's kappa of 0.67.
• 31.1% of the 9591 notes demonstrated high-quality CR documentation.
• CR documentation quality significantly increased with resident postgraduate year (PGY).

Conclusions:

• A high-performing ML model was successfully developed and validated for classifying CR documentation quality in resident admission notes.
• This represents a novel application of ML and NLP in the clinical environment for assessing physician documentation.
• The model has significant potential for improving feedback mechanisms and medical education.