Controlling item difficulty for automatic vocabulary question generation.

Area of Science:

• Educational Technology
• Applied Linguistics
• Natural Language Processing

Background:

• Automatic Question Generation (AQG) systems are increasingly used for creating educational assessments.
• Controlling item difficulty in AQG is crucial for effective language testing.
• Existing AQG systems require robust methods for calibrating difficulty.

Purpose of the Study:

• To identify the most effective factor for controlling the difficulty of multiple-choice English vocabulary questions generated by an AQG system.
• To analyze the impact of reading passage difficulty, distractor-correct answer semantic similarity, and distractor word difficulty on item difficulty.

Main Methods:

• Generated 120 multiple-choice English vocabulary items using an AQG system with varying combinations of three difficulty control factors.
• Administered the machine-generated items to three groups of English learners with different proficiency levels.
• Analyzed the relationship between the manipulated factors and the resulting item difficulty.

Main Results:

• The distractor word difficulty level significantly impacted overall item difficulty.
• The influence of distractor word difficulty on item difficulty varied based on the English learners' proficiency levels.
• Reading passage difficulty and semantic similarity showed less consistent effects on item difficulty.

Conclusions:

• Distractor word difficulty is a primary lever for controlling item difficulty in AQG for English vocabulary.
• AQG systems must adapt difficulty control strategies based on the target learner's proficiency for optimal test administration.
• Findings inform the development of more sophisticated and adaptive automatic testing systems.