Applying the item response theory to classroom examinations

Area of Science:

• Educational Measurement
• Psychometrics
• Health Professions Education

Background:

• Traditional examination analysis may lack precision for smaller class sizes.
• Item Response Theory (IRT) offers advanced psychometric modeling.
• The Rasch Mathematical Model (RMM) is a specific IRT model suitable for smaller datasets.

Purpose of the Study:

• To evaluate the applicability of IRT, specifically the RMM, to small-sized class examinations.
• To assess the robustness of the RMM to missing data in student responses and item removal.
• To determine if students from different examination administrations can be placed on a common metric.

Main Methods:

• The Rasch Mathematical Model (RMM) was employed.
• Two simulated 100-item dichotomous examinations were created for 100 students.
• Simulations included random missing responses and item deletions to test RMM's data robustness.
• Examinations were analyzed individually and combined to compare student scores and IRT measures.

Main Results:

• RMM analysis showed student measures remained unaffected by up to 20% randomly missing responses.
• High reliability and correlation with full data were maintained even with up to 40% of items removed.
• Combining two examinations resulted in highly correlated student scores and IRT measures.

Conclusions:

• The RMM is a viable and successful method for analyzing small-sized class examinations in health professions education.
• Consistent student measurement across different test administrations is achievable with as little as 20% item overlap.
• The RMM can help differentiate between student ability and item difficulty, aiding in score interpretation.