Machine learning strategy identification: A paradigm to uncover decision strategies with high fidelity

We propose a novel approach, which we call machine learning strategy identification (MLSI), to uncovering hidden decision strategies. In this approach, we first train machine learning models on choice and process data of one set of participants who are instructed to use particular strategies, and then use the trained models to identify the strategies employed by a new set of participants. Unlike most modeling approaches that need many trials to identify a participant's strategy, MLSI can distinguish strategies on a trial-by-trial basis. We examined MLSI's performance in three experiments. In Experiment I, we taught participants three different strategies in a paired-comparison decision task. The best machine learning model identified the strategies used by participants with an accuracy rate above 90%. In Experiment II, we compared MLSI with the multiple-measure maximum likelihood (MM-ML) method that is also capable of integrating multiple types of data in strategy identification, and found that MLSI had higher identification accuracy than MM-ML. In Experiment III, we provided feedback to participants who made decisions freely in a task environment that favors the non-compensatory strategy take-the-best. The trial-by-trial results of MLSI show that during the course of the experiment, most participants explored a range of strategies at the beginning, but eventually learned to use take-the-best. Overall, the results of our study demonstrate that MLSI can identify hidden strategies on a trial-by-trial basis and with a high level of accuracy that rivals the performance of other methods that require multiple trials for strategy identification.