A new perspective on detecting performance decline: A change-point analysis based on Jensen-Shannon divergence

Area of Science:

• Psychometrics
• Statistical Modeling
• Educational Measurement

Background:

• Performance decline (PD) in test-takers, characterized by decreased response accuracy on later items, can compromise test validity and ability estimation.
• Existing methods for PD detection often lack sufficient statistical power, typically below 0.55, hindering reliable identification.
• Change-point analysis (CPA) is a statistical technique for detecting abrupt changes in data, but standard methods are not optimized for PD detection.

Purpose of the Study:

• To develop a novel Change-Point Analysis (CPA) method specifically designed for detecting performance decline (PD) in ability assessments.
• To adapt existing CPA statistics into one-sided versions suitable for identifying the directional change associated with PD.
• To evaluate the performance of the proposed PD detection method against modified CPA statistics using simulations and real-world data.

Main Methods:

• Development of a new CPA method utilizing Jensen-Shannon divergence to detect performance decline.
• Conversion of existing CPA statistics to one-sided versions to specifically address the nature of performance decline.
• Conducting a simulation study to compare the power and Type-I error rates of the proposed method against modified CPA statistics.

Main Results:

• The proposed CPA method demonstrated higher statistical power in detecting performance decline compared to modified CPA statistics, with an average increase of 5.7% (ranging from 1.0% to 8.2%).
• The new method maintained a well-controlled Type-I error rate, ensuring reliable detection without excessive false positives.
• The proposed approach also showed improved accuracy in pinpointing the exact location of the performance decline within the test sequence.

Conclusions:

• The developed Jensen-Shannon divergence-based CPA method offers a more powerful and accurate tool for detecting performance decline in ability assessments.
• This advancement addresses the limitations of existing PD detection techniques, contributing to more valid and reliable measurement outcomes.
• The practical utility of the proposed method was confirmed through its successful application to real test datasets.