CoT defender: Preemptive chain-of-thought occupation for jailbreak attack mitigation

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Cybersecurity

Background:

• Large language models (LLMs) are susceptible to jailbreak attacks, compromising their safety and leading to potential abuse.
• Current defense mechanisms often fail to balance robust security with maintaining the usability of LLMs.

Purpose of the Study:

• To introduce CoT Defender, a novel defense strategy to mitigate jailbreak attacks on LLMs.
• To enhance LLM security while preserving the models' usability for benign tasks.

Main Methods:

• CoT Defender employs a chain-of-thought (CoT) analysis in the initial generated tokens to disrupt adversarial inputs.
• A two-stage training framework: Stage 1 fine-tunes for structured CoT, Stage 2 uses reinforcement learning for refinement.
• An auxiliary attacker model generates prompts, and Probabilistic Structured Output Evaluation (PSOE) provides rewards based on intent and format fidelity.

Main Results:

• Reduced average jailbreak attack success rate to below 8.0% across four tested LLMs.
• Maintained high usability, with less than a 7.0% impact on response rates for benign requests.
• Demonstrated effectiveness against six different attack methods.

Conclusions:

• CoT Defender offers an effective solution for enhancing LLM security against jailbreak attacks.
• The proposed method successfully balances robust protection with minimal impact on model usability.
• This research contributes a practical defense mechanism for safer LLM deployment.