A novel approach for assessing fairness in deployed machine learning algorithms.

Area of Science:

• Artificial Intelligence
• Machine Learning Ethics
• Algorithmic Fairness

Background:

• Machine learning (ML) systems increasingly impact societal sectors, raising concerns about fairness.
• Existing research indicates prevalent unfair outcomes in ML applications.
• A statistically validated method to evaluate deployed ML algorithm fairness against datasets is currently lacking.

Purpose of the Study:

• To introduce a novel, statistically validated approach for assessing the fairness of deployed ML algorithms.
• To evaluate the fairness of classical ML algorithms across benchmark datasets using established fairness definitions.
• To investigate the context-dependent nature of ML fairness and its implications.

Main Methods:

• Developed a novel evaluation approach utilizing k-fold cross-validation and statistical t-tests.
• Applied the approach to five benchmark datasets and six classical ML algorithms.
• Considered four distinct fairness definitions from current literature.

Main Results:

• The same dataset yielded fair outcomes for some ML algorithms and unfair outcomes for others.
• Fairness is demonstrated to be a complex issue, highly dependent on the specific ML algorithm and dataset.
• The proposed approach successfully identified variability in fairness outcomes across different ML models.

Conclusions:

• The developed approach enables researchers to statistically assess ML algorithm fairness against protected attributes in datasets.
• Findings underscore the need for adaptable fairness definitions and context-specific evaluations.
• Further research into enhancing ensemble methods for fairness is crucial for equitable AI deployment.