JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models

Area of Science:

• Artificial Intelligence
• Cybersecurity
• Human-Computer Interaction

Background:

• Large language models (LLMs) face security risks from jailbreak attacks, where prompts bypass safety features.
• Evaluating LLM defenses against such attacks and understanding prompt characteristics is complex and time-consuming.

Purpose of the Study:

• To develop an efficient LLM-assisted framework for analyzing jailbreak prompts and assessing LLM security.
• To introduce JailbreakLens, a visual analysis system for exploring LLM performance against jailbreak attacks and prompt characteristics.

Main Methods:

• Collaboration with domain experts to define problem characteristics.
• Development of an LLM-assisted framework for automatic jailbreak assessment and prompt analysis.
• Design of JailbreakLens, a visual system for multi-level prompt analysis and model security evaluation.

Main Results:

• The LLM-assisted framework automates jailbreak assessment and prompt component analysis.
• JailbreakLens enables users to explore model security, analyze prompt features, and refine instances.
• Case studies, technical evaluations, and expert interviews validate the system's effectiveness.

Conclusions:

• The proposed framework and JailbreakLens system significantly streamline the evaluation of LLM security against jailbreak attacks.
• The system aids users in identifying LLM weaknesses and improving model defenses.
• This research contributes to more robust security assessments for large language models.