Effect of attendance at a training session on peer reviewer quality and performance

To determine whether attendance at a voluntary training workshop improves quality ratings of medical journal peer reviewers.

Peer reviewers for Annals of Emergency Medicine who completed two or more reviews during the 20 months before or the 20 months after October 1995 were eligible. Reviews were routinely rated by editors on a subjective 5-point quality scale. Comparisons were made between reviewers who chose to attend a 4-hour workshop on peer review sponsored by the journal in 1995 (attendees) and 2 groups of reviewers who did not attend: controls matched for review quality and number of reviews completed before the workshop, and unmatched controls. Guest reviewers were excluded.

A total of 298 reviewers completed 1906 reviews before the workshop and 2,194 after the workshop; 2,117 of these reviews were rated by editors. Forty-five attendees participated in the workshop, 39 of whom had sufficient ratings for analysis. Matched controls were almost identical in performance to attendees, but unmatched controls had performed fewer reviews and had lower average ratings before the workshop. There was no significant change in any performance measurement after the workshop, including average quality rating, percent change in quality rating, odds ratio for recommending acceptance, and odds ratio for congruence with editor's decision.

In a self-selected group of experienced reviewers who attended a 4-hour workshop on peer review, no effect could be identified in subsequent performance as measured by editors' quality ratings or reviewer performance statistics.