Using Machine Learning Methods to Develop a Short Tree-Based Adaptive Classification Test: Case Study With a...

Area of Science:

• Psychometrics and Educational Measurement
• Machine Learning Applications
• Developmental Psychology

Background:

• Traditional lengthy instruments pose challenges in assessments, especially with high-dimensional data and low-prevalence outcomes.
• Item Response Theory (IRT)-based adaptive testing may struggle with instruments measuring numerous constructs and imbalanced datasets.
• Machine learning offers alternative approaches for developing efficient adaptive tests.

Purpose of the Study:

• To develop and evaluate short tree-based adaptive classification tests using machine learning.
• To assess the feasibility of these tests for a juvenile delinquency risk assessment case study.
• To compare the performance of tree-based adaptive tests against established benchmark methods.

Main Methods:

• A cross-validation study was conducted comparing eight tree-based adaptive test constructions.
• Benchmark methods included IRT scoring and random forest with balanced sampling.
• Data from 3,975 subjects were used, with a focus on imbalanced training data due to low outcome prevalence.

Main Results:

• The best tree-based adaptive tests demonstrated superior classification accuracy compared to IRT scoring.
• Tree-based adaptive tests achieved comparable or better accuracy than random forest with balanced sampling.
• These adaptive tests reduced instrument length by over 30-fold, administering only 3-6 items per individual.

Conclusions:

• Tree-based adaptive tests are a highly effective machine learning approach for shortening instruments with numerous constructs.
• This method shows significant potential for efficient and accurate risk assessment in fields like juvenile delinquency.
• The developed tests offer a practical solution for reducing assessment burden while maintaining diagnostic precision.