Notes From the Field: Automatic Item Generation, Standard Setting, and Learner Performance in Mastery Multiple-Choice...

Area of Science:

• Educational Assessment
• Medical Education Research

Background:

• Mastery learning assessments are established in simulation-based interventions.
• Limited research exists on applying mastery learning to multiple-choice tests (MCTs).

Purpose of the Study:

• To investigate item generation and standard setting for mastery learning MCTs.
• To evaluate the consistency of learner performance across sequential MCTs.

Main Methods:

• Utilized a consensus process to establish item models, variables, and mastery standards.
• Developed two test forms using item models and administered them across two training programs.
• Assessed test-retest consistency of pass-fail decisions and item-level consistency.

Main Results:

• Achieved 94% test-retest consistency in pass-fail decisions (κ = .54).
• Reported a decision-consistency classification of .85.
• Demonstrated 90% item-level consistency (κ = .77).

Conclusions:

• Automatic item generation can create effective mastery learning MCTs.
• This method yields consistent pass-fail decisions for serial testing.
• Expands assessment options for educators using mastery learning curricula.