Auditing the AI auditors: A framework for evaluating fairness and bias in high stakes AI predictive models

Area of Science:

• Psychology and Artificial Intelligence (AI) Ethics

Background:

• Growing concerns regarding unfairness and bias in AI decision-making tools across research, government, and public sectors.
• Exclusion of psychological research from AI fairness discussions due to interdisciplinary terminology and goal mismatches.
• Need for a unified framework to integrate psychological expertise into AI bias and fairness evaluations.

Purpose of the Study:

• To establish a shared interdisciplinary understanding of AI fairness and bias.
• To introduce psychological audits as a standardized approach for evaluating AI systems.
• To provide a comprehensive framework for assessing AI fairness and bias from multiple disciplinary perspectives.

Main Methods:

• Identification of three key disciplinary lenses for examining AI fairness and bias: individual attitudes, legality/ethicality/morality, and technical domain meanings.
• Development of psychological audits as a standardized methodology for AI system evaluation.
• Detailed outline of 12 crucial audit components across AI models, information presentation, and meta-considerations.

Main Results:

• A structured approach (psychological audits) for evaluating AI fairness and bias is presented.
• The audit framework encompasses AI model specifics, stakeholder perspectives, and essential meta-components.
• Integration of psychological principles into AI evaluation is facilitated.

Conclusions:

• Psychological audits offer a standardized, interdisciplinary method to assess and mitigate bias in AI systems.
• The proposed framework enhances the evaluation of AI fairness by incorporating psychological expertise.
• This approach aims to bridge disciplinary gaps and promote more equitable AI development and deployment.