Related Experiment Video
Updated: Jun 28, 2025
09:12
Three Laboratory Procedures for Assessing Different Manifestations of Impulsivity in Rats
Published on: March 17, 2019
9.5K
Inconsistencies in rater-based assessments mainly affect borderline candidates: but using simple heuristics might
Stefan K Schauber1,2, Anne O Olsen3, Erik L Werner4
1Centre for Health Sciences Education, Faculty of Medicine, University of Oslo, Oslo, Norway. stefanks@uio.no.
Advances in Health Sciences Education : Theory and Practice
|April 22, 2024
Summary
Expert judgment in assessments is inconsistent, especially with lower performances. Simple strategies may improve rater consistency and decision-making in educational evaluations.
Area of Science:
- Medical Education
- Psychology
- Cognitive Science
Background:
- Expert judgment exhibits significant inconsistency across various domains.
- In medical education, examiner disagreement in assessments can have serious repercussions.
- Existing literature suggests rating inconsistencies may correlate with candidate performance levels, an area needing further statistical investigation.
Purpose of the Study:
- To investigate the relationship between rater consistency and candidate performance levels in assessments.
- To explore the utility of simple heuristics in enhancing decision-making among examiners.
Main Methods:
- Recorded mock-examinations for analysis.
- Ten examiners evaluated four student performances.
- Bootstrapping procedures were employed to compare inconsistencies in performance ratings between examiner pairs.
Main Results:
- Observed a significant dependence of rating discrepancies on candidate performance levels.
- Lower-performing candidates received more inconsistent ratings compared to excellent performances.
- Analysis suggested that simple heuristics could potentially improve decision-making within examiner pairs.
Conclusions:
- Confirmed the link between rater inconsistencies and candidate performance in assessments.
- Highlighted the importance of considering performance levels to address judgment variability.
- Advocated for the adoption of ecological rationality principles and simple heuristics to improve assessment practices and future research directions.
Related Concept Videos
01:29Reliability and Validity
12.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
12.7K
02:45Self-Discrepancy Theory
18.3K
One influential perspective on what motivates people's behavior is detailed in Tory Higgin's self-discrepancy theory (Higgins, 1987). He proposed that people hold disagreeing internal representations of themselves that lead to different emotional states.
18.3K
01:08The Availability Heuristic
6.0K
A heuristic is a general problem-solving framework (Tversky & Kahneman, 1974). You can think of these as mental shortcuts that are used to solve problems. Different types of heuristics are used in different types of situations, and the impulse to use a heuristic occurs when one of five conditions is met (Pratkanis, 1989):
6.0K
01:21Heuristics
84
Heuristics are problem-solving strategies that use mental shortcuts to simplify decision-making. Unlike algorithms, which must be followed precisely to achieve a correct result, heuristics offer a general problem-solving framework. They save time and energy but can sometimes lead to less rational decisions.
People often rely on heuristics when faced with an overload of information, limited time, low importance of the decision, limited information, or when a heuristic readily comes to mind. For...
People often rely on heuristics when faced with an overload of information, limited time, low importance of the decision, limited information, or when a heuristic readily comes to mind. For...
84
01:14Fundamental Attribution Error
12.8K
According to some social psychologists, people tend to overemphasize internal factors as explanations—or attributions—for the behavior of other people. They tend to assume that the behavior of another person is a trait of that person, and to underestimate the power of the situation on the behavior of others. They tend to fail to recognize when the behavior of another is due to situational variables, and thus to the person’s state. This erroneous assumption is...
12.8K
01:31Confirmation Biases
5.5K
The confirmation bias is the tendency to focus on information that confirms our existing beliefs and ignore information that is inconsistent with our expectations. For example, if you think that your professor is not very nice, you notice all of the instances of rude behavior exhibited by the professor while ignoring the countless pleasant interactions he is involved in on a daily basis. Have you ever fallen prey to the confirmation bias, either as the source or target of such bias?
5.5K