The FRCR 2B oral examination: is it reliable?

To investigate the reliability of the oral component of the Fellowship of the Royal College of Radiologists (FRCR) 2B examination.

Anonymized candidate test scores were analysed from nine consecutive sittings of the FRCR 2B oral examination covering the period from spring 2006 to spring 2010. Interobserver reliability was assessed using weighted Kappa coefficient, intraclass correlation coefficient, and a modified Bland-Altman plot.

During the study period, 2235 candidates sat the FRCR 2B examination. Eighty-five point one percent of candidates obtained paired oral assessment scores within one mark of each other. This figure rises to 95.7% for paired scores within 1.5 marks of each other. Mean difference in scores was 0.67 (95% CI: 0.65-0.70). Agreement rises at the extremities of mean score. Reliability coefficients for the FRCR 2B oral examination were calculated as 0.27 (weighted Kappa) and 0.44 (intraclass correlation coefficient).

The calculated reliability coefficients indicate fair to, at best, moderate interobserver reliability in the FRCR 2B oral examination. These findings are disappointing but are comparable with other oral assessment reliability studies. There is scope for improvement, although further work to measure the combined reliability of all the components of the FRCR 2B examination is desirable. Measures that could potentially increase reliability must be carefully considered against any negative impact on test validity, acceptability, and cost.