CodeSafetyBench evaluating and improving ethical safety in large language model code generation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Software Engineering

Background:

• Large language models (LLMs) are increasingly used in software development, introducing significant ethical and safety concerns.
• Research indicates a high probability (78.67%) of harmful code generation when LLMs receive malicious prompts.

Purpose of the Study:

• To develop a comprehensive benchmark for evaluating the safety of AI-generated code.
• To create and test a novel training framework to mitigate safety risks in LLMs for code generation.

Main Methods:

• Developed CodeSafetyBench, a benchmark with 1,050 harmful request cases across healthcare, law, and education.
• Employed a triadic response structure (harmful outputs, refusals, educational feedback) for training.
• Utilized supervised fine-tuning (SFT) and direct preference optimization (DPO) for model training.

Main Results:

• The developed framework reduced harmful response rates by up to 48%.
• Educational feedback proved more effective in mitigating harmful outputs than simple rejection.
• Demonstrated the effectiveness of the triadic response structure in enhancing AI safety.

Conclusions:

• AI-generated code presents fundamental safety challenges that require systematic solutions.
• The CodeSafetyBench benchmark and SFT/DPO training framework provide a foundation for safer, ethically aligned AI models.
• Balancing high performance with human values is crucial for the responsible development of AI in software engineering.