Individual Differences in Strategy and the Item-Position Effect in Reasoning Ability Measures

Despite the high similarity of reasoning ability items, research indicates that individuals apply different strategies when solving them. The two distinct strategies are response elimination and constructive matching. The latter, frequently showing a positive correlation with reasoning ability, entails the individual systematically investigating the presented problem matrix of an item before scanning the response alternatives. To further understand the sources of individual differences in strategy use during test taking, three different eye-tracking metrics were investigated in participants (N = 210) solving the Raven's Advanced Progressive Matrices (APM). Relying on the fixed-links modeling approach, bifactor models were fit to the data. The latent model approach revealed, in line with other research, a positive correlation between reasoning ability and constructive matching. The results further indicated that a change in strategy use was correlated with the item-position effect and not reasoning ability. The former exhibited a different direction of effect, depending on the eye-tracking metric analyzed. When investigating the toggle rate, the participants used more constructive matching towards the end of the APM. The proportional time to first fixation on response alternatives indicated less constructive matching as the test progressed, and the proportional time on the problem matrix exhibited no distinct pattern regarding a change in strategy use. These diverging results point towards the possibility of a more nuanced problem-solving behavior than previously assumed. By including the item-position effect in the analyses, the increasing individuals differences in problem-solving behavior can be taken into account, which could be a necessary step in attaining a more comprehensive understanding of problem-solving behavior.