Ready to ROC? A tutorial on simulation-based power analyses for null hypothesis significance, minimum-effect, and...

Area of Science:

• Psychological research
• Statistical analysis
• Methodology

Background:

• Receiver operating characteristic (ROC) curves and area under the curve (AUC) are key for assessing discriminability in psychological research.
• Current methods for power analysis in ROC/AUC are often insufficient for complex research questions.

Purpose of the Study:

• To provide a tutorial for simulation-based power analyses of ROC and (p)AUC in R.
• To introduce the "ROCpower" R package and a Shiny app for conducting these analyses.
• To highlight the importance of the smallest effect size of interest (SESOI) in power analysis.

Main Methods:

• Simulation-based power analysis using R.
• Development of the "ROCpower" R package and a Shiny application.
• Focus on a confidence interval-centered approach for power analysis.

Main Results:

• Demonstration of simulation-based power analysis for ROC/AUC in R.
• Guidance on establishing and utilizing the SESOI for various hypothesis testing frameworks (NHST, minimum-effect, equivalence testing).
• The proposed method enhances reproducibility and adaptability across research designs.

Conclusions:

• Simulation-based power analysis, particularly with a focus on SESOI, offers a flexible and reproducible approach for ROC/AUC research in psychology.
• The "ROCpower" package and app facilitate these advanced power analyses.
• This methodology supports a shift towards practically and theoretically relevant effect sizes.