A laboratory study on the reliability estimations of the mini-CEX

Area of Science:

• Medical Education
• Assessment and Evaluation
• Psychometrics

Background:

• Workplace-based assessments, such as the mini-Clinical Evaluation Exercise (mini-CEX), are crucial for evaluating resident performance.
• Traditional reliability estimations often rely on real-life data, which can be confounded by factors like patient variability and assessor bias.
• The assumption of local independence in measurement is difficult to achieve in practice, leading to potential inaccuracies in reliability estimates.

Purpose of the Study:

• To estimate the reproducibility of the mini-CEX using a controlled experimental setup.
• To disentangle the sources of variance contributing to measurement error in the mini-CEX.
• To provide evidence-based recommendations for optimizing mini-CEX reliability.

Main Methods:

• A fully crossed, two-facet generalizability design was employed with 21 residents and 3 assessors.
• Each resident's three encounters were videotaped with identical patients across all residents.
• All assessors evaluated all encounters for all residents, allowing for a systematic analysis of variance.

Main Results:

• The general error term accounted for the largest proportion of variance (34%), followed by the main effect of assessors (18%).
• Universe score variance accounted for 28% of the total variance.
• Generalizability coefficients suggested that 9 encounters are needed for reliable estimation in typical practice, with fewer encounters required when using multiple assessors.

Conclusions:

• Unexplained general error and assessor leniency/stringency are primary contributors to mini-CEX unreliability.
• The findings highlight the need for strategies to mitigate these sources of error.
• Rater training is proposed as a potential intervention to enhance the reliability of the mini-CEX.