[Correcting the critical reading test of the national ranking examination]

In 2008 the national ranking examination (NRE) will include a test on critical reading of scientific articles. This decision has provoked controversy about whether reproducible correction is possible. The aim of our study was to assess the consistency of grading between this two-part test (critical analysis and summarizing, analyzed separately), and the more classic tests.

Eight graders, all instructors at the Tours Medical School, corrected papers for each of the 3 tests. Papers for the critical reading test came from medical school final examinations and for the standard test from a practice examination. The instructors worked in pairs: each pair read 30 papers for each test, and both members separately graded each paper. The final grade was the mean of the two grades. The consistency of grading between the 4 pairs was estimated by Kendall's coefficient of concordance.

Kendall's coefficients of concordance were estimated at 0.94 (95% CI=[0.86; 0.97]) for the standard test, at 0.92 (95% CI=[0.81;0.97]) for the critical analysis test, and at 0.75 (95% CI=[0.62; 0.84]) for the summaries. Pairwise comparisons estimated the difference in concordance between the standard test and the summary at 0.18 (95% CI=[0.08; 0.32]) and that between the standard and critical reading test at 0.01 (95% CI= [-0.07; 0.12]). Finally the difference in concordance between the 2 new tests - summary and critical reading - was estimated at -0.17 (95% CI=[-0.32; -0.04]).

The focus should be on the difficulty of reproducible correction of the summaries, to set up methods for appropriate correction and adequate grading. The elaboration of detailed scoring templates, including numerous items and specifying in which part of the summary each item must be placed, should help to improve the reproducibility of this test's correction.