Decision-making bias in assessment: the effect of aggregating objective information and anecdote

Assessment decisions increasingly rely on synthesis of information from a variety of sources. It is known that aggregation of information to make decisions is open to a number of biases. The aim of this research was to investigate bias, accuracy and confidence of assessment decision making.

The participants were consultation skills assessors. A model for incremental information was developed with participants being shown results from purposefully selected, but authentic, data from the University's final summative 10-station Objective Structured Clinical Examination (OSCE). After each piece of information, participants gave a pass-fail decision and their confidence in that choice. Following the information from 10 OSCE stations the participants were given a discordant fictional anecdote and again participants gave a pass-fail decision and their confidence.

When there is overwhelming evidence to support a pass or fail, participants were not as confident as the data would support. Participants were less confident to make a fail decision than a pass. Despite considerable evidence from multiple results some participants altered decisions based on isolated contradictory information from an anecdote.

These findings are significant in understanding decision-making. Given equivalent levels of evidence, decision makers are less confident to fail than pass and less robust information can undermine more robust information.