Concordance Indices for Risk Scores With Policy Evaluations

Area of Science:

• Health Services Research
• Biostatistics
• Transplantation Science

Background:

• Concordance index (C-index) is a common metric for evaluating risk prediction models in healthcare.
• Existing C-index methodologies may have deficiencies, particularly with dependent censoring common in health services applications.

Purpose of the Study:

• To compare different concordance index (C-index) methodologies.
• To clarify the appropriate use of C-index metrics for risk score evaluations in health services.
• To assess the impact of C-index choice on liver transplant policy evaluations.

Main Methods:

• Methodological comparison of various C-index metrics.
• Analysis of adult liver transplant registry data (Organ Procurement and Transplantation Network) from 2002-2022.
• Evaluation of risk scores using different weighting schemes.

Main Results:

• Gerds' weighted C-index was higher for the original Model for End-Stage Liver Disease (MELD) than other C-index metrics (0.864 vs. 0.854, 0.832, 0.727).
• The concordance metric did not significantly increase after the latest MELD formula update.
• Naïve binary outcome metrics showed substantially lower concordance (0.727).

Conclusions:

• Commonly used concordance indices in health services have critical limitations with dependent censoring.
• Researchers must employ appropriate weighting schemes to prevent biased risk score evaluations.
• Findings necessitate re-interpreting past evaluations that influenced national liver transplant policies.