The Rater Applied Performance Scale: development and reliability.

Area of Science:

• Clinical psychology
• Psychometrics
• Healthcare quality assessment

Background:

• Traditional rater reliability studies often use passive tasks like rating videotaped interviews.
• Limited understanding exists regarding rater performance and scoring consistency during live patient assessments.
• There is a recognized need for systematic methods to evaluate actual rater performance in clinical settings.

Purpose of the Study:

• To introduce and validate the Rater Applied Performance Scale (RAPS) for measuring rater performance.
• To assess the psychometric properties of the RAPS in quantifying rater performance during assessment sessions.
• To explore the utility of the RAPS in improving the quality and consistency of clinical evaluations.

Main Methods:

• The Rater Applied Performance Scale (RAPS) was developed to assess six dimensions: adherence, follow-up, clarification, neutrality, rapport, and accuracy.
• The RAPS was tested by having two independent reviewers evaluate 20 audiotaped Hamilton Depression Scale rating sessions from a multi-site trial.
• Correlations were examined between RAPS ratings and objective measures of scoring accuracy determined by expert raters.

Main Results:

• The RAPS demonstrated good internal consistency among its measured dimensions.
• Satisfactory interrater reliability was achieved for the total RAPS scores.
• RAPS ratings showed a significant correlation with quantitative measures of scoring accuracy, supporting its validity.

Conclusions:

• Preliminary psychometric data indicate that the RAPS is a valuable instrument for quantifying clinical rater performance.
• The RAPS offers a systematic approach to monitoring and improving the quality of assessment sessions.
• Potential applications include training, quality assurance, and research in clinical assessment settings.