Education Research: unsatisfactory NEX rating correlations: searching for the reasons

To determine whether the previously demonstrated poor correlation between local faculty and external American Board of Psychiatry and Neurology (ABPN) examiners evaluating the Neurology Evaluation Exercise (NEX) is attributable to a difference between raters who know the residents and raters who do not, a difference between raters with ABPN experience and raters without it, or some other factor.

Deidentified NEX encounters were videotaped at 2 neurology residency programs. Each video was graded by 1 local faculty examiner, 1 external faculty examiner with ABPN experience, and 1 external faculty examiner without ABPN experience, using the ABPN-sanctioned form. Acceptable/unacceptable rates were compared using Cohen κ statistic.

Fifty-eight videotaped NEX encounters involving 20 residents were evaluated by 12 local faculty examiners, 13 ABPN examiners, and 10 external non-ABPN examiners. The level of agreement between groups failed to meet our target κ of 0.7 (ABPN vs non-ABPN external examiners: κ = 0.47 [95% confidence interval 0.21-0.73]; local vs non-ABPN external examiners: κ = 0.37 [95% confidence interval 0.08-0.66]; local vs ABPN external examiners: κ = 0.40 [95% confidence interval 0.14-0.67]). Local, non-ABPN, and ABPN examiners assigned a failing grade to 13 (22%), 11 (19%), and 16 (28%) of the NEX encounters, respectively.

The disappointing correlation between local examiners, non-ABPN external examiners, and ABPN external examiners is not solely attributable to bias toward familiar residents. Inadequate training in NEX administration and scoring could be a factor. It is also possible that the NEX is not a valid tool. Further study is necessary.