Are Efficient Designs Used in Discrete Choice Experiments Too Difficult for Some Respondents? A Case Study Eliciting...

Although efficient designs have sample size advantages for discrete choice experiments (DCEs), it has been hypothesised that they may result in biased estimates owing to some respondents using simplistic heuristics.

The main objective was to provide a case study documenting that many respondents choose on the basis of a single attribute when exposed to highly efficient DCE designs but switch to a conventional multi-attribute decision rule when the design efficiency was lowered (resulting in less need to trade across all attributes). Additional objectives included comparisons of the sizes of the estimated coefficients and characterisation of heterogeneity, thus providing evidence of the magnitude of bias likely present in highly efficient designs.

Five hundred and twenty-five respondents participating in a wider end-of-life survey each answered two DCEs that varied in their design efficiency. The first was a Street and Burgess 100% efficient Orthogonal Main Effects Plan design (2(7) in 8), using the top and bottom levels of all attributes. The second DCE comprised one eighth of the full Orthogonal Main Effects Plan in 32 pairs, (a 2 × 4(6)). Linear probability models estimated every respondent's complete utility function in DCE1. The number of respondents answering on the basis of one attribute level was noted, as was the proportion of these who then violated this rule in DCE2, the less efficient DCE. Latent class analyses were used to identify heterogeneity.

Sixty per cent of respondents answered all eight tasks comprising DCE1 using a single attribute; most used the rule "choose cheapest end-of-life care plan". However, when answering the four less efficient tasks in DCE2, one third of these (20% overall) then traded across attributes at least once. Among those whose decision rule could not be described qualitatively, latent class models identified two classes; compared to class one, class two was more concerned with quality rather than cost of care and wished to die in an institution rather than at home. Higher efficiency was also associated with smaller regression coefficients, suggesting either weaker preferences or lower choice consistency (larger errors).

This is the first within-subject study to investigate the association between DCE design efficiency and utility estimates. It found that a majority of people did not trade across attributes in the more efficient design but that one third of these then did trade in the less efficient design. More within-subject studies are required to establish how common this is. It may be that future DCEs should attempt to maximise some joint function of statistical and cognitive efficiency to maximise overall efficiency and minimise bias.