Probability of superiority for comparing two groups of clusters

Area of Science:

• Statistics
• Psychometrics
• Educational Research

Background:

• The probability of superiority (PS) is a valuable effect size for comparing two independent samples.
• Existing methods for PS calculation are limited in clustered data contexts common in educational and psychological interventions.
• There is a need for reliable PS estimation methods in clustered designs.

Approach:

• Propose a novel method for estimating the probability of superiority (PS) in clustered data.
• The method applies to study designs comparing two groups where membership is cluster-determined.
• Utilizes nonparametric point estimates and cluster-robust variance estimation for distributional flexibility.

Key Points:

• The proposed method is unbiased for continuous and binary outcomes.
• Demonstrates superior performance compared to the single existing method via Monte Carlo simulation.
• Maintains adequate frequentist properties across various data distributions.

Conclusions:

• The new PS estimation method is suitable for clustered data, including longitudinal designs.
• The approach is easily implementable in standard statistical software with provided R code.
• Aims to improve group difference estimation for researchers using clustered designs.