Impact of ERP Reliability Cutoffs on Sample Characteristics and Effect Sizes: Performance-Monitoring ERPs in...

In studies of event-related brain potentials (ERPs), it is common practice to exclude participants for having too few trials for analysis to ensure adequate score reliability (i.e., internal consistency). However, in research involving clinical samples, the impact of increasingly rigorous reliability standards on factors such as sample generalizability, patient versus control effect sizes, and effect sizes for within-group correlations with external variables is unclear. This study systematically evaluated whether different ERP reliability cutoffs impacted these factors in psychosis. Error-related negativity (ERN) and error positivity (Pe) were assessed during a modified flanker task in 97 patients with psychosis and 104 healthy comparison participants, who also completed measures of cognition and psychiatric symptoms. ERP reliability cutoffs had notably different effects on the factors considered. A recommended reliability cutoff of 0.80 resulted in sample bias due to systematic exclusion of patients with relatively few task errors, lower reported psychiatric symptoms, and higher levels of cognitive functioning. ERP score reliability lower than 0.80 resulted in generally smaller between- and within-group effect sizes, likely misrepresenting effect sizes. Imposing rigorous ERP reliability standards in studies of psychotic disorders might exclude high-functioning patients, which raises important considerations for the generalizability of clinical ERP research. Moving forward, we recommend examining characteristics of excluded participants, optimizing paradigms and processing pipelines for use in clinical samples, justifying reliability thresholds, and routinely reporting score reliability of all measurements, ERP or otherwise, used to examine individual differences, especially in clinical research.