Teaching AI to handle exceptions: Supervised fine-tuning with human-aligned judgment.

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Cognitive Science

Background:

• Large language models (LLMs) are transitioning from generative tasks to agentic decision-making in complex environments.
• The decision-making processes of LLMs, particularly their handling of exceptions, remain poorly understood.
• Strict adherence to policies by LLMs can lead to impractical or suboptimal decisions, diverging from human judgment.

Purpose of the Study:

• To investigate how LLMs handle exceptions in decision-making.
• To evaluate different tuning methods for improving LLM exception handling and human alignment.
• To determine the impact of human explanations versus labels in supervised fine-tuning.

Main Methods:

• Evaluating LLM decision-making against human judgments on exception handling.
• Comparing three tuning approaches: ethical framework prompting, chain-of-thought (CoT) reasoning, and supervised fine-tuning.
• Analyzing the effectiveness of supervised fine-tuning with and without human explanations.

Main Results:

• LLMs deviate from human judgments due to rigid policy adherence, even when impractical.
• Ethical framework prompting was ineffective; CoT prompting offered minor improvements.
• Supervised fine-tuning, especially with human explanations, significantly enhanced LLM decision-making and alignment.
• Fine-tuning with explanations enabled generalization of human-like decision-making to novel scenarios.

Conclusions:

• Aligning LLMs with human judgment requires training on the reasoning process ('how') not just the outcome ('which').
• Supervised fine-tuning with human explanations is critical for developing adaptable and human-aligned agentic AI.
• Addressing LLMs' exception-handling deficits is crucial for advancing AI that effectively aligns with human values and adapts to new contexts.