A hierarchical deep learning framework with doubly regularized loss for robust malware detection and family...

Area of Science:

• Cybersecurity
• Artificial Intelligence
• Machine Learning

Background:

• Increasing cyber threats necessitate advanced malware detection.
• Traditional deep learning methods struggle with class imbalance and overfitting.
• Need for mathematically grounded and interpretable detection frameworks.

Purpose of the Study:

• Propose a hierarchical deep learning framework for Portable Executable (PE) malware detection and family categorization.
• Introduce a novel Doubly Regularized Binary Cross-Entropy (DRBCE) loss function.
• Address challenges like class imbalance, overfitting, and error propagation.

Main Methods:

• Developed a two-stage hierarchical deep learning framework.
• Utilized a novel Doubly Regularized Binary Cross-Entropy (DRBCE) loss function integrating weighted cross-entropy and spectral regularization.
• Modeled error propagation using a probabilistic framework and optimized hyperparameters via constrained optimization.
• Conducted experiments on large-scale datasets (BODMAS, SOMLAP, CLaMP).

Main Results:

• Achieved state-of-the-art performance in malware detection and family categorization.
• Weighted accuracies reached 99.89% for binary detection and 96.62% for family categorization.
• Demonstrated the DRBCE loss function's effectiveness in stabilizing training and mitigating imbalance.

Conclusions:

• The proposed framework offers robust and interpretable malware detection.
• Successfully bridges mathematical modeling with applied AI in cybersecurity.
• Provides theoretical insights and practical advancements for combating sophisticated malware.