Evidence-based recommendations for designing free-sorting experiments

The card-sorting task is a flexible research tool that is widely used across many of the subfields of psychology. Yet this same great flexibility requires researchers to make several (seemingly arbitrary) decisions in their designs, such as fixing a sufficient number of objects to sort, setting task requirements, and creating task instructions for participants. In the present research, we provide a systematic empirical investigation of the consequences of typical researcher design choices while administering sorting tasks. Specifically, we studied the effects of seven sorting task design factors by collecting data from over 1,000 online participants assigned to one of 36 sorting tasks, as part of a fractional factorial experimental design. Analyses show the effects of the various researcher decisions on the probability that participants would quit the task, the amount of time spent on the task, the number of piles made, and posttask measures such as satisfaction and depletion. Research design recommendations are provided.