General or idiosyncratic item effects: What is the good target for models?

Area of Science:

• Psychometrics
• Cognitive Psychology
• Statistical Modeling

Background:

• Criticisms by Adelman et al. (2013) questioned averaging item response times (RTs) and estimating item variance.
• The core assertion was that item effects contain stable, idiosyncratic components.
• This study addresses the validity and implications of these item effects.

Purpose of the Study:

• To provide supplementary empirical evidence supporting the existence of stable idiosyncratic item effects.
• To clarify the statistical implications of item effects on measurement precision and modeling objectives.
• To differentiate between general and idiosyncratic item effects as targets for modeling.

Main Methods:

• Empirical validation of item effects in response time data.
• Statistical analysis to address implications for measurement precision.
• Theoretical clarification of modeling objectives for different item effect types.

Main Results:

• Supplementary empirical evidence confirms the validity of stable idiosyncratic item effects.
• Statistical considerations demonstrate that arbitrary data precision is achievable with sufficient observations per item.
• Distinguishes between general and idiosyncratic item effects as distinct modeling targets.

Conclusions:

• While idiosyncratic item effects are valid, their implications for measurement and modeling differ from Adelman et al.'s claims.
• Achieving desired measurement precision is a function of sample size, not solely dependent on item effect type.
• Both general and idiosyncratic item effects are valuable for modeling, serving different research objectives.