Do complex psychometric analyses really matter? Comparing multiple approaches using individual participant data from...

Psychometric methods are used to remove underperforming items and reduce error in existing measures, albeit different approaches can produce different results. This study aimed to determine the implications of applying different psychometric methods for clinical trial outcomes.

Individual participant data from 15 antidepressant treatment trials from Vivli.org were analyzed. Baseline (pretreatment) and 8-week (range 4-12 weeks) outcome data from the Montgomery-Asberg Depression Rating Scale were subjected to best-practice factor analysis (FA), item response theory (IRT), and network analysis (NA) approaches. Trial outcomes for the original summative scores and psychometric-model scores were assessed using multilevel models. Percentage differences in Cohen's d effect sizes for the original summative and psychometrically modeled scores were the effects of interest.

Each method produced unidimensional models, but the modified scales varied from 7 to 10 items. Treatment effects (d = 0.072) were unchanged for IRT (10 items), decreased by 1.3%-2.8% (eight-item abbreviated d = 0.070; weighted score d = 0.071) for NA, and increased by 11%-12.5% (seven-item abbreviated model d = 0.081; weighted score d = 0.080) for FA.

IRT and NA yielded negligible differences in effect outcomes relative to original trials. FA increased effect sizes and may be the most effective method for identifying the items on which placebo and treatment group outcomes differ.