Automatic- and Transformer-Based Automatic Item Generation: A Critical Review

This article provides a critical review of conceptually different approaches to automatic and transformer-based automatic item generation. Based on a discussion of the current challenges that have arisen due to changes in the use of psychometric tests in recent decades, we outline the requirements that these approaches should ideally fulfill. Subsequently, each approach is examined individually to determine the extent to which it can contribute to meeting the challenges. In doing so, we will focus on the cost savings during the actual item construction phase, the extent to which they may contribute to enhancing test validity, and potential cost savings in the item calibration phase due to either a reduction in the sample size required for item calibration or a reduction in the item loss due to insufficient psychometric characteristics. In addition, the article also aims to outline common recurring themes across these conceptually different approaches and outline areas within each approach that warrant further scientific research.