Enhancing intrusion detection in encrypted DoH traffic through a robust ensemble learning framework

Area of Science:

• Cybersecurity
• Machine Learning

Background:

• DNS over HTTPS (DoH) encrypts queries, improving privacy but enabling covert malicious traffic.
• Detecting such encrypted malicious traffic poses significant challenges for network security.

Purpose of the Study:

• To develop a highly accurate and computationally efficient method for detecting malicious tunneling within DoH traffic.
• To propose a stacked ensemble machine learning approach for identifying covert DoH channels.

Main Methods:

• Employed a stacked ensemble model with Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks as base learners and XGBoost as a meta-learner.
• Utilized Recursive Feature Elimination (RFE) with XGBoost for feature selection, reducing dimensionality from 29 to 13/20 features.
• Applied stratified 5-fold cross-validation for robust model training and evaluation.

Main Results:

• Achieved near-perfect performance metrics (e.g., recall, F1-score, AUC-PR/ROC) on both CIRA-CIC-DoHBrw-2020 and DoH-Tunnel-Traffic-HKD datasets.
• The proposed model significantly outperformed individual base classifiers, other ML models, and existing state-of-the-art methods.
• Demonstrated computational efficiency with training times of 1.26s and inference times of 0.129s on the HKD-DoH dataset.

Conclusions:

• The stacked ensemble framework provides a highly accurate and efficient solution for intrusion detection in encrypted DoH traffic.
• The model effectively addresses the challenge of detecting malicious activities masked by DoH encryption.
• The study offers a practical tool for real-world network security applications.