Malware detection using pre-trained transformer encoder with byte sequences

Area of Science:

• Computer Science
• Cybersecurity
• Artificial Intelligence

Background:

• Malicious attacks are evolving, necessitating advanced malware detection methods.
• Neural networks, particularly convolutional neural networks (CNNs), are used for analyzing file byte sequences.
• Transformer-based language models show promise in various domains and can be adapted for cybersecurity tasks.

Purpose of the Study:

• To introduce a novel Transformer-based language model for malware detection using file byte sequences.
• To propose and evaluate new pre-training strategies for Transformer models in this domain.
• To establish a benchmark dataset for pre-training and fine-tuning malware detection models.

Main Methods:

• Developed a Transformer-based language model to process file byte sequences.
• Introduced two novel pre-training strategies: real-or-fake prediction and same-sequence prediction.
• Explored combinations of pre-training strategies including masked language modeling and next-sentence prediction.
• Created a benchmark dataset comprising PDF, HWP, and MS Office files.

Main Results:

• The proposed Transformer-based language model outperformed CNNs in malware detection.
• Achieved macro F1 score improvements ranging from 2.7%p to 11.1%p compared to CNNs.
• Empirical results validate the effectiveness of the novel pre-training strategies.

Conclusions:

• Transformer-based language models offer a superior approach to CNNs for malware detection.
• The developed model and pre-training strategies provide a strong foundation for future malware detection services.
• Future work will focus on developing encoder-based models for processing longer byte sequences.