Tracking with (Un)Certainty.

Area of Science:

• Educational Technology
• Psychometrics
• Computerized Adaptive Learning (CAL)

Background:

• Personalized learning is a key goal in educational technology, driving the development of Computerized Adaptive Learning (CAL) systems.
• The Elo Rating System (ERS) is widely used in CAL to track student ability and item difficulty, but suffers from drawbacks like lack of standard errors and rating variance inflation.
• Three statistical issues have been identified as the cause of these ERS drawbacks.

Purpose of the Study:

• To address the limitations of the Elo Rating System (ERS) in Computerized Adaptive Learning (CAL) systems.
• To introduce a novel tracking system, Urnings, that resolves statistical issues in ERS and provides standard errors.
• To enable statistical inference, such as testing for learning effects, within CAL environments.

Main Methods:

• Developed a new tracking system, Urnings, representing persons and items as urns with green and red marbles.
• Urns are updated via marble exchange after each response, with marble proportions estimating ability or difficulty.
• Compared the Urnings algorithm against the Elo Rating System (ERS) using simulation studies and empirical CAL data.

Main Results:

• The Urnings algorithm successfully addresses the statistical issues inherent in the Elo Rating System (ERS).
• Urnings provides known standard errors, enabling robust statistical inference.
• The proposed method demonstrates advantages over ERS in both simulated and real-world CAL data.

Conclusions:

• The Urnings algorithm offers a statistically sound alternative to the Elo Rating System (ERS) for tracking in CAL.
• This new approach enhances the capabilities of personalized learning systems by allowing for reliable statistical analysis.
• Urnings facilitates more accurate measurement of student ability and item difficulty, supporting effective educational technology development.