A psychometrics of individual differences in experimental tasks

Area of Science:

• Cognitive Psychology
• Individual Differences Research
• Psychometrics

Background:

• Researchers often correlate task performance to identify shared latent processes.
• Observed low correlations between tasks with presumed commonalities are a persistent challenge.
• This may stem from statistical artifacts (e.g., low individual variability) or substantive issues (e.g., inhibition not being a unified concept).

Purpose of the Study:

• To investigate whether low correlations in individual-differences research are due to statistical aggregation or genuine lack of shared processes.
• To propose and validate an alternative analysis method that accounts for trial-by-trial variability.
• To re-evaluate the relationship between Stroop and flanker task inhibition.

Main Methods:

• Proposed an analysis approach that incorporates trial-by-trial variability and inter-individual covariation.
• Utilized common hierarchical models for implementation.
• Applied the method to data from Hedge et al. (2018) comparing Stroop and flanker tasks.

Main Results:

• The proposed method rescues classical psychometric concepts like effect size, reliability, and correlation.
• Bayes-factor analysis provided strong support for a lack of correlation between the Stroop and flanker tasks.
• This indicates that inhibition measured by these tasks may not be a unified construct.

Conclusions:

• Trial-by-trial aggregation can significantly attenuate effect sizes and correlations in cognitive tasks.
• The findings challenge the notion of a unified inhibitory process, suggesting Stroop and flanker inhibition are distinct.
• This research offers a more accurate approach to studying individual differences in cognitive tasks.