Rasch modeling of accuracy and confidence measures from cognitive tests

The use of IRT models has not been rigorously applied in studies of the relationship between test-takers' confidence and accuracy. This study applied the Rasch measurement models to investigate the relationship between test-takers' confidence and accuracy on English proficiency tests, proposing potentially useful measures of under or overconfidence. The Rasch approach provided the scaffolding to formulate indices that can assess the discrepancy between confidence and accuracy at the item or total test level, as well as at particular ability levels locally. In addition, a "disattenuated" measure of association between accuracy and confidence, which takes measurement error into account, was obtained through a multidimensional Rasch modeling of the two constructs where the latent variance-covariance structure is directly estimated from the data. The results indicate that the participants tend to show overconfidence bias in their own cognitive abilities.