Empowering Expert Judgment: A Data-Driven Decision Framework for Standard Setting in High-Dimensional and Data-Scarce...

Area of Science:

• Psychometrics
• Statistical modeling
• Educational measurement

Background:

• Standard setting faces challenges in small-sample, high-dimensional data where item count exceeds examinee count.
• Traditional parametric models like item response theory can be unstable or fail due to unreliable parameter estimation in these contexts.

Purpose of the Study:

• To investigate and evaluate information-theoretic and unsupervised clustering methods for standard setting in challenging data conditions.
• To establish an evidence-based framework for selecting appropriate data-driven standard-setting methods.

Main Methods:

• A Monte Carlo simulation was employed to systematically evaluate 15 data-driven methods.
• Simulation factors included sample size, item-to-examinee ratio, mixture proportions, item quality, and ability separation.
• Performance was assessed using Relative Error, Classification Accuracy, Sensitivity, Specificity, and Youden's Index.

Main Results:

• No single method proved universally superior; optimal method selection is contingent on examinee mixture proportions.
• The quantile information ratio (QIR) method showed high specificity in scenarios with a dominant non-competent group.
• Clustering methods like Calinski-Harabasz index (CHI) and sum of squared error (SSE) were effective in selective contexts with balanced groups.
• Bayesian kernel density estimation (BKDE) demonstrated robust and balanced performance across various conditions.

Conclusions:

• The study provides a decision framework for practitioners to select defensible, data-driven standard-setting methods when traditional approaches are infeasible.
• The findings highlight the importance of considering examinee characteristics and data structure when choosing a standard-setting methodology.