Generalizability Theory I: Assessing reliability of observational data in the communication sciences

Area of Science:

• Speech-language pathology
• Behavioral sciences
• Psychometrics

Background:

• Perceptual rating scales are vital for evaluating speech outcomes in research and clinical settings.
• The accuracy of these scales depends on observer perceptions, necessitating an optimal number of raters.
• Generalizability Theory provides a framework for assessing reliability in observational data.

Purpose of the Study:

• To review the concepts of Generalizability Theory for application in speech-language pathology.
• To determine the minimum number of raters required for reliable speech outcome evaluations.
• To offer a new interpretation of Generalizability Theory analysis results for clinicians.

Main Methods:

• Review of Generalizability Theory principles.
• Application of Generalizability Theory to speech naturalness data from stuttering treatment.
• Statistical analysis focusing on reliability and variance components.
• Exploration of appropriate and inappropriate uses of the theory.

Main Results:

• Generalizability Theory can guide the determination of the optimal number of raters for speech evaluations.
• Analysis of speech naturalness data provides insights into rater reliability.
• The study highlights statistical considerations and potential pitfalls in applying the theory.
• A novel interpretation of results is proposed for practical clinical application.

Conclusions:

• Generalizability Theory is a valuable tool for optimizing rater numbers in speech perception research.
• The study provides practical guidance for speech-language pathologists using perceptual scales.
• Careful application and interpretation of Generalizability Theory are essential for valid results.