The role of number of items per trial in best-worst scaling experiments

Best-worst scaling is a judgment format in which participants are presented with K items and must choose the best and worst items from that set, along some underlying latent dimension. Best-worst scaling has seen recent use in natural-language processing and psychology to collect lexical semantic norms. In such applications, four items have always been presented on each trial. The present study provides reasoning that values other than 4 might provide better estimates of latent values. The results from simulation experiments and behavioral research confirmed this: Both suggest that, in the general case, six items per trial better reduces errors in the latent value estimates.