Tackling imbalanced data in cybersecurity with transfer learning: a case with ROP payload detection.

Area of Science:

• Cybersecurity
• Machine Learning
• Deep Learning

Background:

• Deep learning models are increasingly popular in cybersecurity due to their effectiveness and generalizability.
• Imbalanced data is a common challenge in cybersecurity, significantly degrading deep learning model performance.
• Traditional machine learning methods often require more human effort and offer less generalizability compared to deep learning.

Purpose of the Study:

• To introduce a novel transfer learning-based method for handling imbalanced data in cybersecurity.
• To apply and evaluate this method for the specific case of return-oriented programming (ROP) payload detection.
• To demonstrate the effectiveness of the proposed method in scenarios with limited or no benign training data.

Main Methods:

• A transfer learning approach was employed to leverage knowledge from source domain programs.
• The method was applied to detect return-oriented programming (ROP) payloads in target domain programs.
• Evaluation was conducted on 3 different target domain programs using 2 different source domain programs, with zero benign training samples in the target domain.

Main Results:

• Achieved an average false positive rate of 0.0290 and an average F1 score of 0.9705.
• Obtained an average detection rate of 0.9521 across different target programs.
• Reduced the total number of false positives by 23.16% compared to the baseline, with a minor decrease of 0.68% in detected malicious samples.

Conclusions:

• The proposed transfer learning method effectively tackles imbalanced data issues in cybersecurity deep learning applications.
• The approach demonstrates strong performance in ROP payload detection, even with limited benign training data.
• A favorable trade-off between reducing false positives and maintaining a high detection rate was observed.