Benefit-Risk or Risk-Benefit Trade-Offs? Another Look at Attribute Ordering Effects in a Pilot Choice Experiment

Studies recommend randomising the order of attributes in discrete choice experiments (DCEs) to avoid bias; however, in a benefit-risk setting, this may increase the cognitive burden of respondents who compare the benefits and risks of treatments, or may affect their decision-making process. Based on these concerns, this paper explored attribute ordering effects in a benefit-risk DCE.

Attribute ordering effects were explored in a large pilot DCE relating to the medical treatment of insomnia. Participants were randomised to one of three presentation orders: (1) benefits were presented before risks (BR); (2) risks were presented before benefits (RB); (3) all attributes were randomised (RN). For the RB and BR presentation orders, attributes were randomised within benefits and risks. Responses were assessed in three ways. First, variations in respondents' self-reported choice certainty were obtained. Second, variations in failure rates of stability and dominance tests were calculated. Third, a heteroscedastic error component model tested for differences in choice consistency across the three attribute orderings.

The final analysis included 156 respondents (RN: 54; BR: 49; RB: 53). No differences were found between the presentation orders with respect to stated choice certainty, or the proportion of respondents failing either the dominance or stability test. However, deterministic attribute grouping was associated with higher choice consistency.

To increase choice consistency, DCE attributes should be randomised within logical groups that may be further randomised to reduce the risk of ordering effects.