Mitigating sensitive information leakage in LLMs4Code through machine unlearning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Software Engineering

Background:

• Large Language Models for Code (LLMs4Code) excel at code generation but risk leaking sensitive training data.
• Existing privacy measures for LLMs4Code are insufficient to prevent sensitive information disclosure.

Purpose of the Study:

• This study provides the first comprehensive empirical analysis of machine unlearning for mitigating sensitive data leakage in LLMs4Code.
• To evaluate the effectiveness of machine unlearning in reducing privacy risks while maintaining model performance.

Main Methods:

• A dedicated benchmark was created with synthetic "forget" and "retain" datasets to test privacy and functionality.
• Three machine unlearning algorithms (GA, GA+GD, GA+KL) were systematically assessed on three open-source LLMs4Code models (AIXCoder-7B, CodeLlama-7B, CodeQwen-7B).

Main Results:

• Machine unlearning reduced direct data leakage by over 50% on average.
• Code-generation performance was retained at over 91% after unlearning.
• A shift from direct to indirect data leakage was observed post-unlearning, indicating a persistent vulnerability.

Conclusions:

• Machine unlearning is a viable and effective method for enhancing privacy in LLMs4Code.
• Future research must develop techniques to address both direct and indirect leakage simultaneously for robust privacy protection.