The interactive reading task: Transformer-based automatic item generation.

Area of Science:

• Natural Language Processing
• Educational Technology
• Psychometrics

Background:

• Traditional automatic item generation (AIG) is limited to easily modeled content areas.
• Developing high-quality assessment items requires significant expert input and time.
• Existing AIG methods struggle with complex construct assessment.

Purpose of the Study:

• To introduce and evaluate an automated approach for creating reading comprehension assessments.
• To leverage deep language models for passage and question generation.
• To demonstrate the feasibility of scalable, construct-driven item development.

Main Methods:

• Developed an interactive reading task using transformer-based deep language modeling.
• Automated the generation of source passages and diverse comprehension questions.
• Utilized selected response question formats for high-fidelity automatic scoring.
• Conducted a large-scale pilot study with hundreds of passages and thousands of questions.

Main Results:

• The interactive reading task successfully generated numerous passages and questions.
• Automated scoring demonstrated high fidelity for selected response formats.
• Human review confirmed the quality and relevance of generated materials.
• Psychometric analyses indicated the approach's feasibility for educational assessments.

Conclusions:

• The interactive reading task offers a viable method for automated creation of reading comprehension assessments.
• This approach expands the scope of AIG to complex content areas.
• It enables a more construct-driven and scalable item development process.