CIPHER: Cybersecurity Intelligent Penetration-Testing Helper for Ethical Researcher

Area of Science:

• Cybersecurity
• Artificial Intelligence
• Penetration Testing

Background:

• Penetration testing is time-consuming and requires specialized knowledge.
• Beginners often need expert guidance for effective vulnerability discovery.
• Existing AI models lack domain-specific training for cybersecurity tasks.

Purpose of the Study:

• To develop a specialized AI chatbot, CIPHER, for penetration testing assistance.
• To create a novel benchmark for evaluating AI in penetration testing.
• To address the limitations of general large language models in cybersecurity.

Main Methods:

• Trained CIPHER on over 300 penetration testing write-ups and tool documentation.
• Introduced the Findings, Action, Reasoning, and Results (FARR) Flow augmentation for automated simulation.
• Established a benchmark for evaluating AI technical knowledge and reasoning in pentesting.

Main Results:

• CIPHER demonstrated superior performance in providing accurate penetration testing suggestions.
• Outperformed similar-sized open-source models and larger state-of-the-art models like Llama 3 70B.
• Highlighted the insufficiency of general LLMs for effective penetration testing guidance.

Conclusions:

• Specialized AI models like CIPHER are crucial for advancing penetration testing.
• The FARR Flow augmentation provides a robust benchmark for AI evaluation.
• Further research into scaling and benchmark development is recommended for AI in cybersecurity.