Value crucible for evaluating robustness of value attributed LLM response profiles via agent adversarial debates

Area of Science:

• Artificial Intelligence
• Computational Social Science
• Psychology

Background:

• Static benchmarks inadequately assess large language models' (LLMs) value-attributed responses in dynamic interactions.
• Understanding response consistency under conversational pressure is crucial for reliable AI behavior.
• Existing evaluations lack frameworks for dynamic, role-conditioned, and adversarial testing.

Purpose of the Study:

• Introduce Value Crucible, a novel agent-based framework to evaluate the robustness of LLM value-attributed responses.
• Assess how role conditioning and adversarial conversational pressure influence LLM value consistency.
• Provide a scalable method for identifying LLM susceptibility to conversational reframing.

Main Methods:

• Developed a three-stage agent-based framework, Value Crucible, for dynamic LLM evaluation.
• Utilized Schwartz's refined value theory and Bardi and Goodwin's dual-route account of value change.
• Evaluated 11 LLMs across 10 social roles and 57 value scenarios, employing self-confrontation debates and the Stance Preservation Index (SPI).

Main Results:

• Observed significant differences in response robustness across LLMs, roles, and value dimensions.
• Found that initially moderate value ratings exhibited greater post-debate shifts compared to strongly endorsed or rejected ratings.
• Demonstrated that response shifts align with circumplex-consistent trade-offs within the Schwartz value space.

Conclusions:

• Value Crucible extends LLM evaluation from static outputs to dynamic, interactive scenarios.
• The framework identifies specific conditions under which LLM value-attributed responses remain robust or shift due to conversational influence.
• This research offers a scalable approach to understanding and improving the consistency of LLM ethical and value-based reasoning.