Interrater reliability in grading abstracts for the orthopaedic trauma association

Area of Science:

• Orthopaedic Surgery
• Medical Education
• Scientific Publishing

Background:

• Selecting abstracts for podium presentations at the Orthopaedic Trauma Association (OTA) annual meeting is competitive.
• Ensuring unbiased and transparent abstract selection is crucial for investigators.

Purpose of the Study:

• To assess reviewer consistency in grading OTA abstracts for podium presentations in 2001 and 2002.
• To evaluate if podium presentation grades align with initial abstract-only grades.

Main Methods:

• Reviewers independently and blindly graded submitted abstracts.
• Abstracts from panel members were adjudicated by non-panel reviewers.
• Reviewers convened to discuss abstracts before final program decisions.
• Agreement was assessed for both abstract grading and full podium presentations.

Main Results:

• Inter-reviewer reliability for abstract grading was low (0.23 in 2001, 0.27 in 2002).
• Consensus was reached through discussion despite initial disagreements.
• Blinded review of podium presentations did not enhance inter-reviewer agreement.
• A significant number of top-ranked presentations were initially graded lower based on abstracts alone.

Conclusions:

• Initial abstract grading by reviewers shows limited consistency.
• Consensus discussions are vital for program selection.
• Abstract grading may not accurately predict the quality of a full podium presentation.
• The selection process may benefit from further refinement to improve accuracy and transparency.