Abstract scoring for the annual SMR program: significance of reviewer score normalization

Presently, the scores of three to four reviewers of each abstract are averaged and form a major component of the input for decisions made in choosing papers for oral and poster presentations, and in rejecting others. No normalization is made for differences in either mean or standard deviation between the reviewers of the same abstracts. In this paper, several techniques for normalizing the scores of reviewers are examined, and the consequences of applying such normalizations to several categories of the 1994 abstract submissions are examined. It was found that some alterations in the acceptance and assignments of papers as oral and posters resulted for one of the categories, and the normalized scores were used during the program assembly. It is recommended that future review procedures utilize reviewer normalization.