Comparing masked and traditional visual analysis of multiple-baseline designs

Area of Science:

• Behavioral Science
• Research Methodology
• Psychometrics

Background:

• Traditional visual analysis (TVA) is standard for single-case experimental designs.
• Concerns about Type I error rates in TVA have led to the development of masked visual analysis (MVA).
• Previous MVA research primarily used simulated data.

Purpose of the Study:

• To evaluate the performance and reliability of MVA using real data and human raters.
• To compare MVA with TVA in the context of single-case experimental designs.
• To assess the validity of MVA in practical research settings.

Main Methods:

• Thirty-six visual analysts evaluated nine graphs from multiple-baseline-design-across-participants studies.
• Graphs were presented in both masked (MVA) and unmasked (TVA) formats.
• Interclass correlation coefficients (ICC) were calculated to assess reliability for both MVA (0.625) and TVA (0.579).

Main Results:

• Reliability for MVA (ICC = 0.625) and TVA (ICC = 0.579) were comparable to previous findings.
• Rater decisions corresponded between MVA and TVA for at least 75% of graphs in six out of nine cases.
• The study demonstrated the feasibility of applying MVA with real-world data.

Conclusions:

• MVA is a reliable method for analyzing single-case experimental designs with real data.
• MVA demonstrates potential for controlling Type I errors without significantly compromising analytic consistency.
• Further research is recommended on MVA and other analytic methods for single-case designs.