Monte Carlo studies of bootstrap variability in ROC analysis with data dependency.

Area of Science:

• Statistics
• Biostatistics
• Machine Learning

Background:

• Receiver Operating Characteristic (ROC) analysis is crucial for classifier performance evaluation across disciplines.
• Data dependency, arising from repeated subject use, is common due to resource limitations.
• Estimating standard errors for ROC statistics with dependent data requires specialized methods.

Purpose of the Study:

• To determine the optimal number of bootstrap replications for ROC analysis with data dependency.
• To reduce bootstrap variance and enhance computational accuracy.
• To provide guidance for robust statistical inference in ROC analysis.

Main Methods:

• A two-layer data structure was employed to handle data dependency.
• Nonparametric two-sample two-layer bootstrap was utilized for standard error estimation.
• Monte Carlo studies assessed bootstrap variability to find appropriate replication numbers.

Main Results:

• The study investigated bootstrap variability in ROC analysis with dependent data.
• A specific number of bootstrap replications was identified as appropriate.
• A tolerance of 0.02 for the coefficient of variation was used.

Conclusions:

• 2,000 bootstrap replications are suggested for ROC analysis with data dependency.
• This recommendation ensures accuracy and reduces variance in statistical computations.
• The findings support reliable classifier evaluation in resource-constrained scenarios.