Using Internal Inconsistency To Identify Unreliable Ratings

Area of Science:

• Psychometrics
• Clinical Psychology
• Behavioral Assessment

Background:

• Clinical ratings are crucial for patient assessment but can be subject to unreliability.
• Identifying and mitigating unreliable ratings is essential for accurate diagnosis and treatment planning.
• Existing methods may not sufficiently address rater-induced inconsistencies in behavioral scales.

Purpose of the Study:

• To develop and validate an internal inconsistency scale for identifying unreliable clinical ratings.
• To differentiate between rater inconsistencies and patient-related inconsistencies.
• To enhance the reliability of factor scores by removing unreliable ratings.

Main Methods:

• Development of an internal inconsistency scale using logically inconsistent item pairs.
• Combination of rational and statistical procedures for item selection and scoring.
• Testing the scale's ability to measure rater inconsistencies and its impact on factor score reliabilities.

Main Results:

• The scale effectively measures rater inconsistencies, not patient inconsistencies.
• Eliminating unreliable ratings based on inconsistency scores improves factor score reliabilities.
• Systematic increase in inconsistency scores with an increased number of items for random assignment.

Conclusions:

• The developed internal inconsistency scale is a valuable tool for identifying unreliable clinical ratings.
• The methodology is adaptable for use with other rating scales beyond the Missouri Inpatient Behavior Scale (MIBS).
• Improving rater consistency enhances the overall reliability and validity of clinical assessments.