Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human...

Area of Science:

• Artificial Intelligence
• AI Ethics
• Sociotechnical Systems

Background:

• Current AI alignment methods, including Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF), aim to align AI systems with human values.
• Large Language Models (LLMs) are a primary focus for these alignment techniques.

Purpose of the Study:

• Critically evaluate the effectiveness of RLHF and RLAIF in aligning AI with human values and intentions.
• Examine the limitations of the 'helpful, harmless, and honest' (HHH) principle in AI alignment.
• Identify neglected ethical considerations in AI alignment research.

Main Methods:

• Multidisciplinary sociotechnical critique of RLHF theoretical underpinnings and practical implementations.
• Analysis of the inherent tensions within the HHH principle.
• Discussion of ethical trade-offs in AI alignment.

Main Results:

• RLHF and RLAIF demonstrate significant shortcomings in capturing the complexities of human ethics.
• The HHH principle presents inherent tensions that complicate AI alignment.
• Ethical issues such as user-friendliness vs. deception and flexibility vs. interpretability are often overlooked.

Conclusions:

• A broader, sociotechnical approach to AI safety and ethics is necessary, integrating institutional, process, and technological design.
• AI safety should be established as a sociotechnical discipline acknowledging normative and political dimensions.