Impact of blinded versus unblinded abstract review on scientific program content

A procedure whereby reviewers are not informed of the author or institutional identity for submitted abstracts is sometimes considered a more equitable and impartial process for selection of the content for a scientific program. We performed a prospective randomized study to evaluate the impact of a reviewer blinding process on scientific program content.

A total of 234 abstracts submitted for presentation at the 2001 meeting of the Southeastern Section of the American Urological Association were distributed for review and grading to 42 reviewers who were randomly assigned to either a blinded or unblinded category. Acceptance for presentation was based on combined raw scores for blinded and unblinded reviews. However, multiple statistical comparisons were performed to evaluate the program content if only the blinded or unblinded reviews had been used as criteria for program acceptance. The abstracts were divided into pediatrics; prostate cancer, bladder cancer and urinary diversion, female urology and urodynamics, endourology and laparoscopy, benign and malignant diseases of the kidney, and penis/erectile dysfunction and miscellaneous for review.

Statistically significant differences were observed in the variability of scores for blinded versus unblinded reviewers in the bladder cancer and prostate cancer groups but not in the other categories. For the pediatrics and urodynamics/female urology groups unblinded reviewers had statistically significant higher mean scores than blinded reviewers but this was not observed in other categories. Overall there was no clear pattern showing greater or less variability between reviewers for blinded versus unblinded reviews, nor was there a consistently observed difference in mean raw scores. Of the papers included in the program 60% would have been accepted for presentation by either blinded or unblinded review alone. This figure increased to 80% when considering the papers which were most highly scored by reviewers in each category.

Although this study cannot be used to determine the quality of reviews based on blinding versus unblinding, it does not demonstrate any consistent increase in variability between reviewers whether they have been informed of abstract authors and institution. Furthermore, there was no identifiable tendency for blinded or unblinded reviewers to assign higher or lower raw scores to abstracts. However, blinded versus nonblinded review may have a substantial impact on program content. Only 60% of the presented abstracts would have been chosen by either method used alone. Additionally, 20% of the 5 abstracts in each category graded most highly by 1 review process would not have been accepted for inclusion on the program by the other process. These data may not be applicable to other circumstances, but the blinding process for abstract review may impact substantially on program content for section meetings.