Investigating the impact of vulnerability datasets on deep learning-based vulnerability detectors

Area of Science:

• Computer Science
• Software Engineering

Background:

• Software vulnerabilities pose significant security risks, driving research in automated detection.
• Deep Learning (DL) methods offer promising approaches for vulnerability detection, reducing manual feature engineering and improving accuracy.
• The effectiveness of DL models is highly dependent on the characteristics of the vulnerability datasets used for training.

Purpose of the Study:

• To systematically characterize vulnerability datasets beyond simple statistics.
• To analyze the correlation between dataset characteristics (sample similarity, code features, granularity) and DL-based vulnerability detector performance.
• To provide insights into how dataset properties impact DL model effectiveness.

Main Methods:

• Developed methods to characterize sample similarity and code features within vulnerability datasets.
• Utilized sample granularity, sample similarity, and code features as key dataset characterization metrics.
• Analyzed the relationship between these characteristics and the performance of DL-based vulnerability detectors.

Main Results:

• Fine-grained samples in datasets are beneficial for vulnerability detection.
• Datasets with lower inter-class similarity and higher intra-class similarity, along with simpler structures, enhance detection on the original test set.
• Datasets with higher inter-class similarity and lower intra-class similarity, featuring complex structures, show better generalization to other datasets.

Conclusions:

• Dataset characteristics significantly influence the performance of Deep Learning-based vulnerability detectors.
• Understanding and characterizing vulnerability datasets is crucial for developing robust and effective detection systems.
• Tailoring dataset properties can optimize DL models for specific vulnerability detection tasks.