Assessing communication skills during OSCE: need for integrated psychometric approaches

Area of Science:

• Medical Education
• Health Professions Education
• Communication Skills Assessment

Background:

• Physicians' communication skills (CS) are crucial for quality healthcare.
• CS training is integral to medical curricula and assessed via Objective Structured Clinical Examinations (OSCE).
• Reliable instruments are essential for evaluating these communication skills.

Purpose of the Study:

• To test the factorial validity and reliability of a four-item global rating scale for measuring CS in medical students.
• To assess the scale's performance in Objective Structured Clinical Examinations (OSCEs).

Main Methods:

• Retrospective study utilizing Exploratory Factor Analysis (EFA), Multi-Group Confirmatory Factor Analysis (MGCFA), and Item Response Theory (IRT).
• Analysis of data from 296 third- and fourth-year medical students at the Faculty of Medicine in Geneva, Switzerland.
• Validation of a four-item scale developed by Hodges and McIlroy.

Main Results:

• Exploratory Factor Analysis (EFA) indicated good reliability scores at individual stations.
• Multi-Group Confirmatory Factor Analysis (MGCFA) revealed unsatisfactory measurement invariance across stations and student groups.
• Item Response Theory (IRT) showed that the four items primarily captured high levels of communication skills, with overlapping information.

Conclusions:

• The current four-item scale may struggle to differentiate between students with poor and better communication skills.
• Future assessments should consider training examiners for score consistency and evaluating items for broader spectrum coverage.
• Item Response Theory (IRT) is valuable for ongoing evaluation of CS measurement instruments in performance-based assessments.