Comparing Holistic and Mixed-Approach Rubrics for Academic Poster Quality

Area of Science:

• Educational Measurement
• Scientific Communication

Background:

• Poster quality varies significantly in academic settings.
• Existing poster-quality rubrics lack robust psychometric evidence.
• Developing reliable assessment tools is crucial for academic conferences.

Purpose of the Study:

• To compare holistic versus mixed-approach scoring using a new rubric.
• To evaluate the rubric's validation evidence and time-to-score utility.
• To assess the impact of rater experience on scoring reliability.

Main Methods:

• Sixty research posters were scored by four pharmacy education faculty.
• Raters used a new rubric, first holistically, then with a mixed approach (overall + sub-scores).
• Generalizability Theory and Rasch Measurement Model were employed for analysis.

Main Results:

• The mixed approach enhanced scoring reliability, particularly reducing the effect of rater experience.
• Rasch analysis confirmed better rating scale function and construct validation with the mixed approach.
• Holistic scoring was faster, but the mixed approach offered comparable speed with improved reliability.

Conclusions:

• The mixed-approach rubric provides a reliable and efficient method for poster assessment.
• It minimizes the need for rater training and accommodates varying experience levels.
• This rubric is applicable for students, faculty, judges, and potentially other academic presentation formats.