How Accurate and Consistent Are Score-Based Assessment Decisions? A Procedure Using the Linear Factor Model.

Area of Science:

• Psychometrics
• Statistical Modeling
• Psychological Assessment

Background:

• Accurate and consistent decisions are crucial when using scales for individual assessments, such as psychiatric disorder screening.
• Evaluating scale accuracy and consistency is challenging with single-administration data and unknown true conditions.
• Existing methods may not adequately address the complexities of single-administration data in psychological testing.

Purpose of the Study:

• To present a novel method for evaluating the accuracy and consistency of scale-based decisions using data from a single administration.
• To provide practical tools, including R code, for researchers to implement this evaluation method.
• To assess the performance of the proposed method with discrete (vs. continuous) scale items.

Main Methods:

• Development of a method based on the linear factor model.
• Application of the method to data from a single administration of a scale.
• Illustrative examples and R code for practical implementation.
• Simulation study to evaluate performance with discrete items.

Main Results:

• The proposed linear factor model-based method effectively evaluates accuracy and consistency from single-administration scale data.
• The R code facilitates the application of this method in diverse research settings.
• The method demonstrates robustness when applied to scales with discrete items, a common practice.

Conclusions:

• The described method offers a reliable approach to assessing the quality of decisions derived from psychometric scales.
• Researchers can confidently use this method, even with commonly used discrete-item scales.
• This work enhances the rigor of psychological assessment and decision-making processes.