Related Experiment Video
Updated: Jul 6, 2026
08:12
A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments
Published on: March 1, 2022
Testing intergroup concordance in ranking experiments with two groups of judges
Dawn J Dekle1, Denis H Y Leung, Min Zhu
1School of Economics, Singapore Management University, Singapore.
Psychological Methods
|March 12, 2008
Summary
This study compares methods for measuring agreement between groups when judges come from multiple groups. Using actual ranks is better than pairwise comparisons, and missing data significantly reduces statistical power.
Area of Science:
- Psychology
- Statistics
Background:
- Concordance measures intragroup agreement in ranking items.
- Assessing intergroup concordance requires understanding within-group agreement.
Purpose of the Study:
- Compare existing methods for measuring intergroup concordance.
- Evaluate method performance under various scenarios, including missing data.
Main Methods:
- Utilized permutation theory to calculate error rates and statistical power.
- Investigated scenarios with varying numbers of items, within-group agreement, and between-group agreement.
- Studied the impact of missing data and imputation methods.
Main Results:
- Method performance is contingent on the number of items, and within- and between-group agreement levels.
- Directly using item ranks outperformed pairwise ranking comparisons.
- Missing data substantially decreased statistical power, influenced by the missing data mechanism and imputation technique.
Conclusions:
- The choice of concordance measurement method is critical and depends on specific study parameters.
- Handling missing data appropriately is crucial to maintain statistical power in concordance studies.
Related Concept Videos
01:21Friedman Two-way Analysis of Variance by Ranks
Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures from...
01:20Kendall's Coefficient of Concordance
Kendall's Coefficient of Concordance (W), also known as Kendall's W, is a non-parametric statistical measure used to assess the agreement or concordance between multiple raters or judges when they rank a set of items. It is often used when you have ordinal data (ranks) and you want to see if there is consistency or consensus among the raters. It is widely applied in research areas such as psychology, medicine, and social sciences, where multiple judges are asked to rank or rate subjects or...
01:02Ranks
Unlike parametric methods, nonparametric statistics are ideal for nominal and ordinal data, requiring fewer assumptions about the population's nature or distribution. This makes nonparametric methods easier to apply and interpret, as they do not depend on parameters like mean or standard deviation. One common approach in nonparametric analysis is to sort data according to a specific criterion. For instance, we might arrange weather data from hottest to coldest days in a month or rank cities...
01:23McNemar's Test
McNemar's Test is a nonparametric statistical test used to determine if there is a significant difference in proportions between two related groups when the outcome is binary (e.g., yes/no, success/failure). It is beneficial when we have paired data, such as pre-test/post-test designs, where the same subjects are measured under two different conditions. The test is named after the statistician Quinn McNemar, who introduced it in 1947. It is commonly used in situations where subjects are...
01:10Bonferroni Test
The Bonferroni test is a statistical test named after Carlo Emilio Bonferroni, an Italian mathematician best known for Bonferroni inequalities. This statistical test is a type of multiple comparison test to determine which means are different than the rest. Bonferroni test can minimize the Type 1 error by reducing the significance level alpha, which otherwise increases with sample pairs.
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
01:20Comparing the Survival Analysis of Two or More Groups
Survival analysis is a cornerstone of medical research, used to evaluate the time until an event of interest occurs, such as death, disease recurrence, or recovery. Unlike standard statistical methods, survival analysis is particularly adept at handling censored data—instances where the event has not occurred for some participants by the end of the study or remains unobserved. To address these unique challenges, specialized techniques like the Kaplan-Meier estimator, log-rank test, and Cox...