Exploring the inequitable impact of data missingness on fairness in machine learning

Today, data-driven models and artificial intelligence / machine learning underlie decision making in almost all aspects of society. However, significant concerns have been raised over the fairness of such models. While various aspects of algorithmic fairness have been studied, the effect of missing data on fairness remains understudied. This is a significant problem since data in real-world settings is almost never complete, and may often suffer from systemic missingness. This article systematically evaluates how missing data, particularly when correlated with protected classes and outcome variables, affects the fairness of classifiers. Utilizing a comprehensive framework covering various missing data patterns, rates, and mitigation methods, we analyze 150 experimental dataset variants derived from real-world scenarios, and find that missing data correlated with sensitive attributes and outcomes can exacerbate disparities, even for little missingness, making it crucial to address missingness in fairness evaluations.