No Free Lunch: Retrieval-Augmented Generation Undermines Fairness in LLMs, Even for Vigilant Users.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning Ethics

Background:

• Retrieval-Augmented Generation (RAG) is a popular technique for enhancing Large Language Models (LLMs).
• RAG improves LLM factuality and domain-specific knowledge without costly retraining.
• Concerns about fairness and potential biases in LLM outputs persist.

Purpose of the Study:

• To investigate the fairness costs and implications of using RAG.
• To analyze how different levels of data censorship in RAG affect fairness outcomes.
• To evaluate the vulnerability of RAG systems to fairness-related attacks.

Main Methods:

• Proposed a three-level threat model based on user fairness awareness.
• Conducted experiments using RAG with uncensored, partially censored, and fully censored external datasets.
• Assessed fairness implications without fine-tuning or retraining the LLMs.

Main Results:

• Fairness alignment in RAG can be easily compromised without model modification.
• Even fully censored and purportedly unbiased external datasets can lead to biased LLM outputs via RAG.
• RAG systems demonstrate susceptibility to fairness undermining.

Conclusions:

• Current LLM alignment methods may be insufficient for RAG systems.
• RAG introduces unique fairness challenges that require novel mitigation strategies.
• Further research is crucial for developing robust fairness safeguards in RAG-based LLMs.