Related Experiment Video
Updated: Jun 9, 2025
07:13
A Two-interval Forced-choice Task for Multisensory Comparisons
Published on: November 9, 2018
10.9K
Reaction-time task reliability is more accurately computed with permutation-based split-half correlations than with
Sercan Kahveci1,2, Arne C Bathke3, Jens Blechert4,5
1Department of Psychology, Paris-Lodron-University of Salzburg, Hellbrunner Straße 34, 5020, Salzburg, Austria. sercan.kahveci@plus.ac.at.
Psychonomic Bulletin & Review
|October 23, 2024
Summary
Cronbach's alpha provides inaccurate reliability estimates for reaction-time tasks. Permutation-based split-half reliability offers a more accurate method for assessing experimental paradigm reliability.
Area of Science:
- Psychological measurement
- Cognitive psychology
- Psychometrics
Background:
- Reliability reporting is standard for self-report scales but uncommon for experimental paradigms.
- Reaction-time tasks are widely used in psychological research, necessitating accurate reliability measures.
Purpose of the Study:
- To review and compare methods for computing reliability in reaction-time tasks.
- To identify the most accurate reliability estimation technique for experimental paradigms.
Main Methods:
- Simulation study comparing Cronbach's alpha and split-half reliability methods.
- Reanalysis of existing data and literature comparison.
- Establishing best practices for permutation-based split-half reliability via simulations.
Main Results:
- Cronbach's alpha yields inaccurate and biased reliability estimates for reaction-time data.
- Permutation-based split-half reliability provides substantially more accurate estimates.
- Optimizing permutation-based split-half reliability involves equal trial distribution per stimulus, Spearman-Brown correction, and sufficient split-half computations (approx. 5,400).
Conclusions:
- Permutation-based split-half reliability is the recommended method for assessing reaction-time task reliability.
- Current practices using Cronbach's alpha can lead to misleading conclusions about measurement precision.
- Guidelines are provided for maximizing the accuracy of permutation-based split-half reliability, with a review of available software.
More Related Videos
Related Concept Videos
01:23McNemar's Test
161
McNemar's Test is a nonparametric statistical test used to determine if there is a significant difference in proportions between two related groups when the outcome is binary (e.g., yes/no, success/failure). It is beneficial when we have paired data, such as pre-test/post-test designs, where the same subjects are measured under two different conditions. The test is named after the statistician Quinn McNemar, who introduced it in 1947. It is commonly used in situations where subjects are...
161
01:10Bonferroni Test
2.7K
The Bonferroni test is a statistical test named after Carlo Emilio Bonferroni, an Italian mathematician best known for Bonferroni inequalities. This statistical test is a type of multiple comparison test to determine which means are different than the rest. Bonferroni test can minimize the Type 1 error by reducing the significance level alpha, which otherwise increases with sample pairs.
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
2.7K
01:20Spearman's Rank Correlation Test
688
Spearman's rank correlation test, also known as Spearman's rho, is a nonparametric method for assessing the strength and direction of association between two variables. This test is particularly valuable when the data distribution is unknown or when the assumption of normality does not hold. Named after the English psychologist and statistician Dr. Charles Edward Spearman, it serves as the nonparametric counterpart to Pearson's correlation coefficient.
Spearman's test calculates...
Spearman's test calculates...
688
01:22Self-Report Tests of Personality
310
Self-report inventories are objective personality assessments that use multiple-choice items or numbered scales, typically ranging from 1 (strongly disagree) to 5 (strongly agree). They are often called Likert scales after Rensis Likert. These inventories are widely used due to their ease of administration and cost-effectiveness. One of the most prominent examples is the Minnesota Multiphasic Personality Inventory (MMPI), initially developed in the 1940s to assess abnormal personality traits.
310
01:13Accuracy and Errors in Hypothesis Testing
176
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
176
01:21Friedman Two-way Analysis of Variance by Ranks
150
Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures...
150