Certainty-Based Marking on Multiple-Choice Items: Psychometrics Meets Decision Theory

Area of Science:

• Educational Psychology
• Psychometrics
• Decision Science

Background:

• Traditional multiple-choice tests cannot distinguish knowledge from uncertainty.
• Certainty-based marking (CBM) incorporates examinee certainty into scoring.
• Prospect theory suggests risk attitudes influence decision-making, potentially impacting CBM.

Purpose of the Study:

• To investigate response behaviors in Certainty-based marking (CBM) among physiotherapy students.
• To integrate psychometric modeling with decision-making theory to analyze CBM.
• To understand how risk attitudes affect certainty reporting in a CBM context.

Main Methods:

• Utilized item response theory to model the probability of correct responses.
• Applied cumulative prospect theory to estimate examinee risk attitudes.
• Conducted a case study with 334 first-year physiotherapy students across six CBM examinations.

Main Results:

• Student certainty levels in CBM were significantly influenced by their risk attitudes.
• Risk-averse and loss-averse students tended to under-report certainty with high success probabilities.
• Risk-seeking students tended to over-report certainty with low success probabilities.

Conclusions:

• Risk attitudes play a crucial role in how students report certainty under CBM scoring.
• The findings highlight a potential bias in CBM due to non-rational decision-making.
• Understanding these behaviors is key to refining CBM for more accurate assessment.