Is Easier Better Than Harder? An Experiment on Choice Experiments for Benefit-Risk Tradeoff Preferences

To test the convergent validity of simple and more complex study designs in a discrete-choice experiment (DCE) of multiple sclerosis (MS) treatment preferences.

Five hundred US adults with MS completed an online DCE survey. Respondents answered 8 choice questions with pairs of constructed MS treatment profiles defined by delays in problems with walking, delays in problems with cognition, thyroid disorders, and 10-y risks of kidney failure and serious brain infection (i.e., progressive multifocal leukoencephalopathy [PML]). Four hundred respondents completed choice questions using 4 levels for all attributes, except thyroid disorders with 3 levels. One hundred respondents completed choice questions using only the 2 extreme attribute levels of the 4-level version. Random-parameters logit models were used to estimate choice-model parameters.

Respondents viewing the 4-level and 2-level versions agreed on the relative importance of the 3 most important attributes: cognition, walking, and PML. Respondents viewing the 4-level version indicated much stronger disutility for a 0% to 0.5% increase in kidney-failure risk than those viewing the 2-level version where the risk for kidney failure increased from 0% to 3%. Otherwise, utilities for other 4-level attributes were approximately linear but with significantly steeper slopes (except for cognition) than the 2-level estimates, indicating that attributes were perceived as more important as the number of levels increased.

Although the relative importance of some attributes was similar, the 2-level and 4-level versions generally failed to demonstrate convergent validity. If the study goal is attribute rankings, a 2-level version could be adequate. If goals include quantifying tradeoffs among attribute levels, more complex designs can help respondents discriminate among attribute levels. Reductions in measurement error using fewer attribute levels appear to have come at the expense of less discriminating evaluations.