Machine learning to extract communication and history-taking skills in OSCE transcripts

Area of Science:

• Medical Education Technology
• Natural Language Processing in Healthcare
• Machine Learning Applications

Background:

• Observed Structured Clinical Exams (OSCEs) are crucial for assessing medical students' clinical skills and providing feedback.
• Current OSCE assessment methods using checklists and global scales have known limitations.
• Automated analysis of OSCE transcripts could offer more objective and efficient evaluation.

Purpose of the Study:

• To apply machine learning (ML) to automatically label communication skills and interview content within OSCE transcripts.
• To compare the performance and transferability of different ML methodologies for analyzing OSCE data.
• To explore the potential of ML for enhancing the accuracy and efficiency of OSCE assessments.

Main Methods:

• Manually annotated 121 OSCE transcripts across 19 communication and content areas.
• Converted utterances into numeric sentence vector representations and applied three ML algorithms.
• Evaluated ML models using K-fold cross-validation for performance (F1 scores) and tested transferability between scenarios.

Main Results:

• ML models achieved high median F1 scores (0.87) in performance testing across 19 labels.
• Successful transferability was demonstrated with a median F1 score of 0.76 on unseen scenario transcripts.
• A bi-directional long short-term memory (biLSTM) neural network with GenSen vectors showed superior performance and transferability.

Conclusions:

• This study demonstrates the first application of ML for analyzing student-standardized patient OSCE transcripts.
• ML models can effectively label OSCE transcripts for communication skills and interview content.
• Optimized ML models offer potential for automated, accurate OSCE assessment, aiding student progress tracking and targeted practice.