Estimating Between-Person and Within-Person Subscore Reliability with Profile Analysis.

Area of Science:

• Educational Measurement
• Psychometrics
• Psychological Testing

Background:

• Subscores are increasingly important for diagnosing examinee strengths and weaknesses.
• Previous research primarily focused on individual subscore reliability, neglecting their distinctiveness and added value over total scores.

Purpose of the Study:

• To introduce a novel profile reliability approach for subscores.
• To partition overall subscore reliability into within-person and between-person components.
• To evaluate the distinctiveness and added value of subscores.

Main Methods:

• Developed and demonstrated a profile reliability approach.
• Estimated within-person and between-person reliability coefficients.
• Utilized simulation and real data studies with various scoring methods (e.g., IRT).

Main Results:

• A significant trade-off exists between within-person and between-person subscore reliability.
• The proposed profile reliability coefficients effectively quantify distinctness under different testing conditions.
• Subtest length, correlations, and number of subtests impact reliability.

Conclusions:

• The profile reliability approach offers a more comprehensive evaluation of subscores.
• Understanding the balance between within-person and between-person reliability is key for test development.
• This method aids in determining the diagnostic utility of subscores in various contexts.