AI-Augmented Content Validation in Behavioral Research: Development and Evaluation of the RATER System

Area of Science:

• Psychometrics
• Behavioral Science
• Artificial Intelligence

Background:

• Content validation is essential for measurement instrument development, ensuring scales accurately capture intended constructs.
• Researchers often skip content validation due to high costs and specialized expertise requirements.
• Existing methods for content validation are resource-intensive, limiting their widespread use in behavioral research.

Purpose of the Study:

• To introduce RATER (Replicable Approach to Expert Ratings), a free, web-based system designed to streamline content validity assessment.
• To provide researchers, reviewers, and students with accessible and reliable tools for evaluating measurement instruments.
• To enhance the scale development and validation process in behavioral research through automated content validity insights.

Main Methods:

• RATER utilizes psychometric measurement theory to evaluate scale items against intended constructs, construct distinctness, and domain representation.
• The system incorporates two artificial intelligence models, RATER_C and RATER_D, trained on 2,443 psychometric scales from diverse disciplines.
• Leverages advanced large language model architectures, including BERT and GPT, for sophisticated analysis.

Main Results:

• Six studies confirmed the RATER system's accuracy, reliability, and practical utility.
• RATER demonstrated its capability to provide quick and dependable insights into the content validity of measurement instruments.
• The system effectively assesses whether scale items align with their intended constructs and represent the domain comprehensively.

Conclusions:

• RATER significantly augments the scale development and validation process by offering an efficient and reliable method for content validation.
• The tool democratizes access to content validity assessment, overcoming previous barriers of cost and expertise.
• Implementing RATER can lead to more valid and robust findings in behavioral research by improving the quality of measurement instruments.