Powerful exact unconditional tests for agreement between two raters with binary endpoints.

Area of Science:

• Statistics
• Biostatistics
• Psychometrics

Background:

• Traditional methods for testing agreement with binary outcomes include asymptotic and exact conditional approaches.
• The exact conditional approach maintains test size better than the asymptotic approach using Cohen's kappa.
• Unconditional strategies offer an alternative by relaxing constraints on marginal totals.

Purpose of the Study:

• To evaluate three exact unconditional hypothesis testing procedures for binary outcomes.
• To compare these procedures against the standardized Cohen's kappa coefficient.
• To identify the most powerful and reliable methods for practical application.

Main Methods:

• Consideration of three exact unconditional hypothesis testing procedures: maximization, conditional p-value and maximization, and estimation and maximization.
• Comparison of these methods with the standardized Cohen's kappa coefficient.
• Evaluation based on test size and statistical power.

Main Results:

• The exact conditional approach guarantees test size validity, outperforming the asymptotic approach.
• Exact unconditional approaches, particularly those involving conditional p-values and estimation with maximization, demonstrate power advantages.
• The study identified specific unconditional procedures as more effective than traditional methods.

Conclusions:

• Exact unconditional hypothesis testing offers a viable alternative to conditional approaches for binary outcome agreement.
• The approaches based on conditional p-value and maximization, and estimation and maximization, are recommended for practical use due to their power.
• These recommended methods provide enhanced statistical power for assessing inter-rater reliability.