The Choice of Task Format can Affect the Quality of Published Best-Worst Scaling Studies

Best-worst scaling (BWS) is a theory-driven method to measure priorities with growing applications in health and other literatures. Priorities in BWS are elicited through a series of choice tasks. While several study design factors have been linked to the overall quality of published BWS studies, there is a paucity of evidence linking the choice of task format on quality. We sought to identify the range of task formats used in published BWS studies, assess how frequently each format was applied, and examine whether task format influences the quality of published BWS studies. We identified the task format of BWS studies published prior to January 1, 2025, via example tasks, descriptions, appendices, and/or supplementary materials. We then developed a simple taxonomy of the most common format types (e.g., best anchor on the left), except for when task format was missing. Quality scores were calculated using the PREFS checklist and then used to test for differences across task formats in a two-sample t-test. Z-scores were used to summarize differences relative to the best anchor on the left, and differences across task format types were examined using one-way ANOVA with pairwise comparisons.Among the 714 published BWS studies identified, over a third were missing information on task format (n = 253). The remaining studies were classified into four types: best anchor on the left, which was both the first format used in a published study and accounted for 62% of studies, both anchors on one side (17%), best anchor on the right (14%), and multiple anchors/tasks (7%). Best anchor on the left had the highest mean PREFS score of 3.41. Studies that did not report task format had significantly lower mean PREFS scores (p < 0.01) compared with studies that did report task format. The 'best anchor on the left' format was the first and most used task format. It was associated with higher study quality relative to less common or unspecified formats. Studies that did not report task format had lower study quality than studies that clearly specified their format, underscoring the importance of transparent task design and reporting in BWS research.