Developing a sentence level fairness metric using word embeddings

Area of Science:

• Computational Linguistics
• Social Psychology
• Natural Language Processing

Background:

• Fairness is a fundamental social value.
• A quantitative metric for assessing fairness in digital text describing social interactions is currently lacking.
• Existing methods do not capture the nuances of fairness in text.

Purpose of the Study:

• To develop a quantitative metric for assessing fairness in digital texts.
• To approximate human perceptions of fairness using computational methods.
• To introduce an ethical dimension to the analysis of digital texts.

Main Methods:

• Utilized word embeddings informed by social psychology literature to construct fairness vectors.
• Applied Universal Sentence Encoder (USE) and Sentence Bidirectional Encoder Representations from Transformers (SBERT) for sentence-level fairness measurement.
• Employed principal component analysis (PCA) and machine learning (ML) techniques.

Main Results:

• Achieved an F1 score of 79.8 with USE and 86.2 with PCA/ML for fairness detection.
• SBERT achieved significantly higher performance, with F1 scores of 96.9 and 100.
• Demonstrated the effectiveness of leveraging pro-social bias in word embeddings.

Conclusions:

• The proposed method provides a robust approach to quantifying fairness in digital text.
• The developed fairness vectors serve as effective sentence-level measures of ethical components.
• This research opens new avenues for ethical analysis in natural language processing.