Assessing Fairness in the Presence of Missing Data

Area of Science:

• Machine Learning and Data Science
• Algorithmic Fairness
• Statistical Analysis

Background:

• Missing data is a common issue in real-world datasets, complicating data analysis and predictive modeling.
• Existing research on algorithmic fairness primarily focuses on fully observed data, leaving a gap in understanding fairness with incomplete data.
• A common approach to handle missing data is complete case analysis, which may lead to biased results due to distributional shifts.

Purpose of the Study:

• To theoretically investigate and estimate fairness in the complete data domain when analysis is performed using only complete cases from incomplete datasets.
• To address the potential for biased predictions towards marginalized groups when fairness is assessed solely on complete cases.
• To provide the first theoretical framework for fairness guarantees in the analysis of incomplete data.

Main Methods:

• Development of theoretical upper and lower bounds for estimating fairness error in the complete data domain.
• Evaluation of an arbitrary prediction model using only complete cases.
• Conducting numerical experiments to validate the theoretical findings.

Main Results:

• The study establishes theoretical bounds on the error of fairness estimation when using complete cases.
• Numerical experiments demonstrate the practical implications of the theoretical results.
• The findings quantify the potential discrepancy in fairness when evaluating models on complete cases versus the complete data.

Conclusions:

• This research provides the foundational theoretical results for ensuring fairness in machine learning models trained on incomplete data.
• The developed bounds offer a way to understand and potentially mitigate fairness estimation errors arising from complete case analysis.
• The work highlights the importance of considering missing data mechanisms when assessing algorithmic fairness.