Biases of success rate differences shown in binomial effect size displays

Area of Science:

• Statistics
• Psychometrics
• Quantitative Psychology

Background:

• The binomial effect size display (BESD) translates correlations (r) into success rate differences (SRDs) to illustrate real-world importance.
• BESD aims to provide an intuitive understanding of effect sizes in practical terms.

Purpose of the Study:

• To evaluate the accuracy of the binomial effect size display (BESD) in representing real-world effect sizes.
• To identify potential overestimation biases in BESD when applied to different correlation coefficients.
• To assess the comparability of BESD-derived SRDs across various statistical associations.

Main Methods:

• The study analyzes the relationship between correlation coefficients (phi, point-biserial [rpbs], and rxy) and the success rate differences (SRDs) they represent.
• It investigates the overestimation biases inherent in the BESD method for different types of correlations.
• The stochastic difference index is examined as a potential alternative measure.

Main Results:

• BESD generally overestimates the true success rate difference (SRD) implied by correlations.
• Overestimation bias is more pronounced for continuous X and Y variables (rxys) compared to dichotomous X and continuous Y (rpbs) or dichotomous X and Y (phi).
• The varying biases indicate that BESD-derived SRDs are not directly comparable across different correlation types.

Conclusions:

• The binomial effect size display (BESD) introduces significant overestimation biases, particularly with continuous variables.
• The lack of comparability across different correlation types limits the universal application of BESD.
• The stochastic difference index is proposed as a more reliable alternative for effect size estimation.