Evolving Diverse Red-team Language Models in Multi-round Multi-agent Games.

Area of Science:

• Artificial Intelligence
• Machine Learning Security
• Natural Language Processing

Background:

• Ensuring the harmlessness of Large Language Models (LLMs) is a primary deployment challenge.
• Current red teaming methods use static, single-round prompts, leading to reduced generation diversity (mode collapse) and limiting risk discovery.
• Complex human-LLM interactions require more dynamic safety evaluation approaches.

Purpose of the Study:

• To introduce a dynamic Red Team Game (RTG) for analyzing multi-round offensive and defensive interactions between red and blue teams.
• To develop a Gamified Red Team Solver (GRTS) to mitigate mode collapse and improve LLM safety strategies.
• To theoretically guarantee the convergence of approximate Nash equilibrium for enhanced adversarial training.

Main Methods:

• Development of a dynamic Red Team Game (RTG) framework.
• Implementation of a Gamified Red Team Solver (GRTS) incorporating diversity measures.
• Theoretical analysis to guarantee convergence of approximate Nash equilibrium.
• Empirical evaluation of GRTS against various LLMs.

Main Results:

• GRTS effectively mitigates mode collapse, exploring diverse and implicit attacks.
• The approach adaptively exploits various LLMs, surpassing limitations of static methods.
• Empirical results demonstrate improved strategy convergence and enhanced LLM safety analysis.
• The study reveals a geometrical structure aligning with the spinning top hypothesis, supporting diverse LLM populations for red teaming.

Conclusions:

• Dynamic Red Team Game (RTG) and Gamified Red Team Solver (GRTS) offer a robust framework for LLM safety evaluation.
• Mitigating mode collapse through diversity measures is crucial for comprehensive risk discovery.
• A diverse LLM population serves as a proxy for heterogeneous human red-teamers, advancing scalable toxicity detection and safe AI alignment.