Assessing the validity of computer-based test interpretations: rating reliability and individual differences among...

Area of Science:

• Psychological assessment
• Psychometrics
• Individual differences research

Background:

• Computer-based test interpretations (CBTIs) are increasingly used in psychological assessments.
• Validation studies of CBTIs rely on rater judgments, but the influence of individual rater differences is not fully understood.
• The Sixteen Personality Factor Questionnaire (16PF) is a widely used personality assessment tool with associated CBTIs.

Purpose of the Study:

• To investigate the impact of individual rater differences on the accuracy and reliability of computer-based test interpretations (CBTIs).
• To assess the test-retest reliability of accuracy and usefulness ratings for two specific CBTIs of the Sixteen Personality Factor Questionnaire (16PF).

Main Methods:

• Three studies were conducted involving student raters evaluating two CBTIs for the 16PF: the Human Resource Development Report (HRDR) and the Narrative Score Report (NSR).
• Raters evaluated report accuracy in experimental and nonexperimental contexts.
• Test-retest reliability was calculated for accuracy and usefulness ratings.

Main Results:

• Specific personality traits (e.g., shrewdness, self-sufficiency, emotional stability) influenced CBTI accuracy ratings, with effects varying by context and report.
• Test-retest reliability for accuracy ratings was high for both HRDR (.74) and NSR (.81).
• Test-retest reliability for usefulness ratings was high for HRDR (.75) but moderate for NSR (.54).

Conclusions:

• Individual rater characteristics play a role in the perceived accuracy of CBTIs, particularly in experimental settings.
• The reliability of accuracy ratings for 16PF CBTIs is generally strong, while usefulness ratings show more variability.
• Findings highlight the importance of considering rater individual differences in the validation of computer-based psychological assessment tools.