Related Experiment Video
Updated: Jun 10, 2026
08:33
A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences
Published on: September 4, 2019
Reliability of performance examinations: revisited
1Measurement Research Associates, Inc., 505 N. Lake Shore Dr., Ste. 1304, Chicago, IL 60611, USA. mlunz@MeasurementResearch.com
Journal of Applied Measurement
|August 10, 2010
Summary
This study shows that performance examination results are reliable. Using the multi-facet Rasch model ensures consistent pass-fail decisions for most candidates when data are collected properly.
Area of Science:
- Psychometrics
- Educational Measurement
- Performance Assessment
Background:
- Assessing the reliability of performance examinations is crucial for valid pass-fail decisions.
- Candidate outcomes in performance assessments can be influenced by various factors like examiners and tasks.
- Standardization is key to ensuring consistent and comparable assessment results.
Purpose of the Study:
- To evaluate the reproducibility of pass-fail decisions in performance examinations.
- To demonstrate how the multi-facet Rasch model enhances reliability in performance testing.
- To establish confidence in candidate outcomes through rigorous data collection and analysis.
Main Methods:
- Application of the multi-facet Rasch model to account for examination facets (examiners, tasks, items).
- Standardization of the examination process to collect sufficient candidate data.
- Linking of different facets within the examination to a common criterion.
Main Results:
- The multi-facet Rasch model effectively measures candidates against a single, consistent criterion.
- Standardized data collection and linked facets lead to high confidence in results.
- A high percentage of candidates demonstrate reproducible outcomes.
Conclusions:
- Performance examinations can achieve high reliability in pass-fail decisions.
- The multi-facet Rasch model provides a robust framework for analyzing performance assessment data.
- Properly collected and analyzed data ensure confidence in the reproducibility of candidate performance.
Related Concept Videos
01:29Reliability and Validity
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
01:29Theory of Attribution II: Kelley's Covariation Theory
Attribution theory plays a crucial role in social psychology, helping to explain how individuals interpret the causes of behavior. One prominent model within this field is Harold Kelley's covariation theory, which provides a systematic approach to determining whether internal traits or external circumstances drive a person's actions. The model posits that individuals rely on three key types of information—consensus, consistency, and distinctiveness—to make these judgments.Consensus: Comparing...
01:24Sources of Self-Esteem II: Performance Feedback
Self-esteem is intricately tied to our perception of competence and our ability to exert control over our lives. One of the primary sources of this perception is performance feedback — the ongoing evaluation of our actions in terms of success and failure. According to Franks and Marolla (1976), people derive self-worth from experiencing themselves as causal agents, capable of achieving goals and overcoming obstacles. This process nurtures a critical component of self-esteem: self-efficacy,...
01:29Measures of Intelligence
Psychologists measure intelligence by using standardized tests that produce a score known as the intelligence quotient or IQ. To understand IQ tests, it's important to recognize the key principles behind their construction: validity, reliability, and standardization.
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this; it...
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this; it...
01:29Self-Evaluation Maintenance Model
The Self-Evaluation Maintenance (SEM) model offers a psychological framework to understand how individuals’ self-esteem is influenced by the achievements of others, particularly those with whom they share close personal bonds. The SEM model operates when personal rather than social identity guides individuals. Central to this model is the notion that individuals have an inherent desire to preserve a favorable self-image, which is continuously shaped by interpersonal comparisons and...
03:37Uncertainty in Measurement: Accuracy and Precision
Scientists typically make repeated measurements of a quantity to ensure the quality of their findings and to evaluate both the precision and the accuracy of their results. Measurements are said to be precise if they yield very similar results when repeated in the same manner. A measurement is considered accurate if it yields a result that is very close to the true or the accepted value. Precise values agree with each other; accurate values agree with a true value.