"Robust" but Tiny: Methodological Influences and Inter-Individual (Un)Stability of the Serial Order Effect in...

Area of Science:

• Cognitive Psychology
• Creativity Research
• Psychometrics

Background:

• The serial order effect (SOE) describes how creative idea quality improves with response position in divergent thinking (DT) tasks.
• Previous research has documented the SOE but rarely quantified its magnitude or examined its methodological sensitivity.
• The stability of individual differences in SOE has remained uninvestigated.

Purpose of the Study:

• To assess the strength and reliability of the SOE across different methods.
• To determine if the SOE is a dependable, trait-like psychological phenomenon.
• To investigate the SOE's consistency across various divergent thinking tasks and scoring techniques.

Main Methods:

• Examined SOE strength and inter-individual consistency using three Alternate Uses Task (AUT) prompts.
• Employed three scoring approaches: subjective ratings, frequency-based scoring, and semantic similarity.
• Analyzed data for linear and quadratic trends and calculated intraclass correlations (ICCs) for stability.

Main Results:

• The SOE was present across all scoring methods and functional forms, but effect sizes were small (≤3.5% variance).
• SOE magnitude varied significantly based on the specific AUT prompt and scoring method used.
• No evidence of stable individual differences in SOE slopes was found (ICCs < 0.31).

Conclusions:

• The SOE demonstrates group-level robustness but lacks inter-individual reliability, presenting a "reliability paradox."
• Findings question the SOE's status as a stable psychological trait and highlight its limited practical significance at the individual level.
• Systematic evaluation of SOE effect sizes and methodological influences is crucial for future research.