Related Experiment Video
Updated: Apr 23, 2026
An Automated System for Sound Localization Testing in Hearing-Impaired Listeners
Published on: March 13, 2026
Audio-digital recordings used for independent confirmation of site-based MADRS interview scores
Steven D Targum1, J Cara Pendergrass1, Chelsea Toner1
1Clintara LLC, Boston, MA, USA.
Abstract:
Signal detection requires ratings reliability throughout a clinical trial. The confirmation of site-based rater scores by a second, independent and blinded rater is a reasonable metric of ratings reliability. We used audio-digital pens to record site-based interviews of the Montgomery-Asberg Depression Rating Scale (MADRS) in a double-blind, placebo controlled trial of a novel antidepressant in treatment resistant depressed patients. Blinded, site-independent raters generated "dual" scores that revealed high correlations between site-based and site-independent raters (r=0.940 for all ratings) and high sensitivity, specificity, predictive values, and kappa coefficients for treatment response and non-response outcomes using the site-based rater scores as the standard. The blinded raters achieved an 89.4% overall accuracy and 0.786 kappa for matching the treatment response or non-response outcomes of the site-based raters. A limitation of this method is that independent ratings depend on the quality of site-based interviews and patient responses to the site-based interviewers. Nonetheless, this quality assurance strategy may have broad applicability for studies that use subjective measures and wherever ratings reliability is a concern. "Dual" scoring of recorded site-based ratings can be a relatively unobtrusive surveillance strategy to confirm scores and to identify and remediate rater "outliers" during a study.
Insights
Ensuring rating reliability in clinical trials is crucial. "Dual" scoring by independent, blinded raters using audio-digital pens confirmed site-based assessments, demonstrating high accuracy and broad applicability for subjective measures.
Area of Science:
- Clinical Psychology
- Psychiatry
- Medical Technology
Background:
- Clinical trial success hinges on reliable data collection, particularly for subjective measures.
- Rater variability can introduce bias and compromise trial integrity.
- The Montgomery-Asberg Depression Rating Scale (MADRS) is a key subjective measure in depression trials.
Purpose of the Study:
- To evaluate the reliability of site-based rater scores using a novel "dual" scoring method.
- To assess the accuracy and feasibility of using independent, blinded raters to confirm clinical trial assessments.
- To explore the utility of audio-digital pens in enhancing data quality for subjective rating scales.
Main Methods:
- Audio-digital pens recorded site-based interviews for the MADRS during a double-blind, placebo-controlled trial.
- Independent, blinded raters generated "dual" scores from these recordings.
- Statistical analysis compared site-based and site-independent ratings for correlation, sensitivity, specificity, and kappa coefficients.
Main Results:
- High correlation (r=0.940) was observed between site-based and site-independent rater scores.
- Blinded raters achieved 89.4% overall accuracy and a 0.786 kappa coefficient in matching treatment response outcomes.
- The "dual" scoring method demonstrated high sensitivity, specificity, and predictive values.
Conclusions:
- "Dual" scoring by independent, blinded raters is a reliable method to confirm subjective assessments in clinical trials.
- This quality assurance strategy enhances data integrity and can identify "outlier" raters.
- The method shows broad applicability for studies utilizing subjective measures and requiring high ratings reliability.
