Use of Crossed Random-Effects Models to Assess Multiple-Choice Items: An Experimental Study
Susana Sanz1,2, Carmen García2, Ricardo Olmos2
1Department of Psychology, https://ror.org/017mdc710Universidad Pontificia Comillas, Spain.
The Spanish Journal of Psychology
|April 29, 2026
Summary
Including None-of-the-above (NOTA) options in tests can negatively impact accuracy and increase item difficulty. Personality traits and prior knowledge also influence how students interact with NOTA, affecting overall test performance.
Area of Science:
- Educational Psychology
- Psychometrics
Background:
- The inclusion of "None of the above" (NOTA) options in assessments may affect psychometric properties.
- Personality traits are linked to academic performance and could influence the impact of NOTA.
Purpose of the Study:
- To investigate the effects of NOTA appearance and usage on test performance.
- To examine the moderating role of personality traits on NOTA effects.
Main Methods:
- Manipulated NOTA appearance and function (correct option vs. distractor).
- Utilized a statistics concept inventory with 449 psychology students.
- Employed a crossed random-effects model to analyze item and student covariates.
Main Results:
- NOTA's appearance negatively impacted the probability of correct answers, with effects varying across students.
- Conscientiousness, Extraversion, Emotional Stability, and prior statistics knowledge were significant predictors.
- NOTA inclusion increased item difficulty, but not all students responded uniformly to NOTA.
Conclusions:
- The presence of NOTA options can increase test item difficulty and alter response probabilities.
- Individual differences, particularly personality traits and prior knowledge, mediate the effects of NOTA on student performance.
Related Concept Videos
01:15Bioequivalence Experimental Study Designs: Repeated Measures, Cross-Over, Carry-Over, and Latin Square Designs
417
Bioequivalence experimental study designs play a pivotal role in testing the effectiveness of various treatments. Key among these are the repeated measures, cross-over, carry-over, and Latin square designs. In the repeated measures design, each subject receives all treatments, allowing for temporal comparisons. This type of design is useful in reducing variability but requires careful planning to avoid bias.The cross-over design, an economical method, involves sequential administration of...
417
01:16Crossover Experiments
3.4K
Crossover experiments, also called the repeated-measurements design, is a study design in which all experimental units are exposed to all treatments in different periods. Crossover experiments are generally used in psychology, the pharmaceutical industry, agriculture, and medicine.
Crossover designs are performed even with smaller sample sizes since the samples can act as their controls. These are better than simple randomized trials since patients are exposed to all the treatments.
Crossover designs are performed even with smaller sample sizes since the samples can act as their controls. These are better than simple randomized trials since patients are exposed to all the treatments.
3.4K
02:01Group Design
9.3K
The most basic experimental design involves two groups: the experimental group and the control group. The two groups are designed to be the same except for one difference— experimental manipulation. The experimental group gets the experimental manipulation—that is, the treatment or variable being tested—and the control group does not. Since experimental manipulation is the only difference between the experimental and control groups, we can be sure that any differences between...
9.3K
01:13Randomized Experiments
6.3K
The randomization process involves assigning study participants randomly to experimental or control groups based on their probability of being equally assigned. Randomization is meant to eliminate selection bias and balance known and unknown confounding factors so that the control group is similar to the treatment group as much as possible. A computer program and a random number generator can be used to assign participants to groups in a way that minimizes bias.
Simple randomization
Simple...
Simple randomization
Simple...
6.3K
01:21Friedman Two-way Analysis of Variance by Ranks
595
Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures...
595
01:13Multiple Comparison Tests
3.4K
Multiple comparison test, abbreviated as MCT, is a post hoc analysis generally performed after comparing multiple samples with one or more tests. An MCT will help identify a significantly different sample among multiple samples or a factor among multiple factors.
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
3.4K