"Cephalgia" or "migraine"? Solving the headache of assessing clinical reasoning using natural language processing

Area of Science:

• Medical Education
• Natural Language Processing
• Health Informatics

Background:

• Clinical reasoning is a crucial medical competency.
• Traditional assessment methods like multiple-choice questions are insufficient for evaluating clinical reasoning.
• Constructed-response assessments offer better measurement but require significant resources.

Purpose of the Study:

• To discuss the benefits of using Natural Language Processing (NLP) for clinical reasoning assessment.
• To introduce the Intelligent Clinical Text Evaluator (INCITE), an NLP-based scoring system.
• To explore the potential of NLP tools for formative assessment.

Main Methods:

• Overview of INCITE, a scalable NLP-based computer-assisted scoring system.
• Design of an NLP pipeline focused on transparency and interpretability.
• Utilizing written documentation from the USMLE Step 2 Clinical Skills examination.

Main Results:

• INCITE provides a scalable solution for assessing clinical reasoning via written text.
• The NLP pipeline's transparency allows for traceability of scores to evaluated text segments.
• The system's design facilitates repurposing for formative assessment.

Conclusions:

• NLP offers a powerful approach to enhance the assessment of clinical reasoning.
• Transparent and interpretable NLP systems like INCITE can improve educational feedback.
• The development of NLP-based assessment tools can support medical education and evaluation.