Comparing Analytic and Mixed-Approach Rubrics for Academic Poster Quality

Area of Science:

• Educational Assessment
• Scientific Communication

Background:

• Rubrics are widely used for assessment, with analytic rubrics (ARs) and mixed-approach rubrics (MARs) being common types.
• Different rubric types possess distinct advantages and disadvantages for evaluating academic work.

Purpose of the Study:

• To compare the effectiveness and efficiency of analytic rubrics (ARs) versus a mixed-approach rubric (MAR) for assessing research poster quality.
• To evaluate inter-rater reliability using both traditional and modern measurement techniques.

Main Methods:

• Two analytic rubrics (AR1, AR2) and one mixed-approach rubric (MAR) were used to score 60 research posters.
• Two independent scorers evaluated each poster, with scoring time recorded.
• Inter-rater reliability was assessed using intraclass correlations and Rasch measurement.

Main Results:

• The MAR, a single-item rubric, effectively distinguished between high and low-quality posters, similar to a 9-item AR2 and better than a 9-item AR1.
• The MAR exhibited better functioning rating scales compared to AR1 and AR2.
• The MAR required less scoring time than both AR1 and AR2.

Conclusions:

• The mixed-approach rubric (MAR) offers comparable or superior measurement accuracy to analytic rubrics (ARs) while being more time-efficient.
• Findings challenge the misconception that ARs are inherently more accurate or the most efficient assessment tools.