Understanding Large Language Model Behaviors Through Interactive Counterfactual Generation and Analysis.

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Explainable AI (XAI)

Background:

• Current explainable AI (XAI) methods for large language models (LLMs) often use inefficient word-level explanations.
• Existing XAI approaches may not align with human reasoning and treat explanations as static outputs.
• There is a need for more intuitive, efficient, and interactive methods to understand LLM behavior.

Purpose of the Study:

• To introduce LLM Analyzer, an interactive visualization system for exploring LLM behaviors.
• To address limitations of current XAI methods by enabling intuitive and efficient counterfactual analysis.
• To facilitate a more active and human-centered approach to AI explanation.

Main Methods:

• Developed a novel algorithm for generating fluent and semantically meaningful counterfactuals through targeted removal and replacement operations.
• Utilized counterfactuals to compute feature attribution scores at user-defined levels of granularity.
• Integrated feature attribution scores with concrete examples in a table-based visualization for dynamic analysis.

Main Results:

• LLM Analyzer enables intuitive and efficient exploration of LLM behaviors via counterfactual analysis.
• The system supports dynamic analysis by integrating feature attribution with concrete examples.
• User studies and expert interviews confirmed the system's usability and effectiveness.

Conclusions:

• LLM Analyzer provides a more effective approach to understanding LLM behavior compared to traditional word-level explanations.
• Interactive visualization and counterfactual analysis are key to improving LLM explainability.
• Emphasizes the importance of active human participation in the AI explanation process.