Automatic analysis of summary statements in virtual patients - a pilot study evaluating a machine learning approach

Area of Science:

• Medical Education Technology
• Natural Language Processing in Healthcare

Background:

• Concise patient summary statements reflect clinical reasoning skills in healthcare students.
• A published rubric assesses these statements manually across five categories.
• Automated assessment of these statements could provide real-time feedback.

Purpose of the Study:

• To explore computer-based methods for automatically assessing student summary statements.
• To evaluate the feasibility of real-time feedback using automated assessment.
• To compare automated ratings with manual rubric-based assessments.

Main Methods:

• 125 German and English summary statements from virtual patient scenarios were randomly selected.
• Statements were manually rated using a rubric, including patient name usage.
• A natural language processing approach and a custom algorithm were used for automated assessment.

Main Results:

• Moderate agreement was found between manual and automated ratings in most categories.
• Further development is required for accurate assessment of factual accuracy and patient name identification in German statements.
• The study identified areas for improvement in automated assessment algorithms.

Conclusions:

• Computer-calculated assessment scores can be cautiously displayed as learner feedback.
• Learners should be informed that ratings are approximations and allowed to contest them.
• Learner feedback and appeals will aid in refining the automated rating algorithms.