Professionalism and clinical short answer question marking with machine learning.

Area of Science:

• Medical Education
• Artificial Intelligence
• Natural Language Processing

Background:

• Medical student evaluation is crucial for professional development.
• Traditional assessment methods can be time-consuming and subjective.
• Objective and scalable evaluation tools are needed.

Purpose of the Study:

• To investigate the efficacy of machine learning (ML) in scoring medical student short answer questions.
• To assess the performance of bidirectional encoder representations from transformers (BERT) in evaluating professionalism.
• To compare ML performance on professionalism versus clinical questions.

Main Methods:

• Short answer questions were administered to medical students across three centers.
• Machine learning models, specifically BERT, were employed for automated scoring.
• Classification accuracy was calculated for professionalism and clinical questions.

Main Results:

• BERT achieved high accuracy (41.6%–92.5%) in scoring professionalism questions.
• Machine learning demonstrated lower classification accuracy for 3-mark professionalism questions compared to clinical questions (P < 0.05).

Conclusions:

• Machine learning, especially BERT, shows potential for assisting in medical student professionalism evaluation.
• Further investigation is warranted to enhance ML accuracy for specific types of professionalism assessments.
• The application of AI in medical education assessment requires continued exploration.