Assessing Ability Recovery of the Sequential IRT Model With Unstructured Multiple-Attempt Data

Area of Science:

• Educational Measurement
• Psychometrics
• Learning Analytics

Background:

• Unstructured multiple-attempt (MA) item response data from virtual learning environments (VLEs) present challenges for educational measurement due to missing data and unknown ability growth.
• Accurate ability measurement from VLE data is crucial for improving VLE systems, monitoring student progress, and supporting educational research.

Purpose of the Study:

• To evaluate the ability recovery of the multidimensional sequential 2-PL item response theory (IRT) model when applied to unstructured MA data from VLEs.
• To investigate the impact of ability growth magnitude and the proportion of students with two attempts on ability estimation accuracy.
• To examine the moderating effects of sample size, test length, and data missingness on the bias and accuracy of ability estimates.

Main Methods:

• A simulation study was designed to assess the performance of the multidimensional sequential 2-PL IRT model.
• The simulation manipulated key parameters including ability growth, student attempt proportions, sample size, test length, and missing data rates.
• Bias and root mean square error (RMSE) of ability estimates were calculated to evaluate model performance under various conditions.

Main Results:

• The multidimensional sequential 2-PL IRT model demonstrates potential for ability estimation in unstructured VLE data.
• Significant bias in ability estimates was observed under specific data conditions, highlighting potential limitations.
• The magnitude of ability growth and the proportion of students with multiple attempts were found to influence estimation accuracy.

Conclusions:

• The multidimensional sequential 2-PL IRT model shows promise for analyzing complex VLE data, but careful consideration of data characteristics is necessary.
• Certain data conditions, such as high missingness or specific growth patterns, can compromise the accuracy of ability estimates.
• Further research is needed to refine models and methods for robust ability estimation from diverse VLE data.