Assessing Algorithm Fairness Requires Adjustment for Risk Distribution Differences: Re-Considering the Equal...

Area of Science:

• Computer Science
• Statistics
• Healthcare Analytics

Background:

• Algorithm-assisted decision-making necessitates robust fairness assessments.
• Equal opportunity, a common fairness metric, relies on true positive rate (TPR) parity across subgroups.
• Existing metrics may misinterpret performance disparities due to varying subgroup risk distributions.

Purpose of the Study:

• To introduce a novel fairness metric, adjusted true positive rate (aTPR), addressing limitations of traditional TPR.
• To ensure fairness evaluation accounts for differential risk distributions across population subgroups.
• To promote equal treatment for individuals with similar underlying risks, irrespective of group affiliation.

Main Methods:

• Developed aTPR metric by normalizing subgroup TPRs against a reference subgroup's risk distribution.
• Conducted numerical experiments to analyze performance under various differential calibration scenarios.
• Applied the aTPR metric to a real-world dataset predicting in-patient mortality risk.

Main Results:

• Demonstrated that standard TPR can yield misleading fairness conclusions when risk distributions differ across subgroups.
• Showcased how aTPR provides a more accurate assessment of fairness by adjusting for baseline risk variations.
• Identified potential performance disparities in palliative care referral predictions using the aTPR metric.

Conclusions:

• The adjusted true positive rate (aTPR) offers a more reliable approach to evaluating algorithmic fairness, especially when subgroup risk distributions vary.
• Accurate fairness assessment is crucial for equitable deployment of algorithms in healthcare and other sensitive domains.
• This work facilitates more equitable risk prediction and resource allocation, such as timely palliative care consultations.