Investigating Generalized Performance of Data-Constrained Supervised Machine Learning Models on Novel, Related...

Area of Science:

• Cybersecurity
• Machine Learning
• Network Intrusion Detection

Background:

• Machine learning methods are increasingly proposed for intrusion detection systems.
• Current validation relies heavily on performance metrics within specific academic datasets.
• Generalizability of these models to unseen data is rarely assessed.

Purpose of the Study:

• To investigate the generalizability of machine learning intrusion detection models across different datasets.
• To determine if high performance on one dataset predicts performance on others.
• To highlight the limitations of current validation practices in intrusion detection research.

Main Methods:

• Trained binary classification models on CSE-CIC-IDS2018 dataset.
• Tested model performance on CIC-IDS2017, CIC-DoS2017, and CIC-DDoS2019 datasets.
• Analyzed classification metrics (precision, recall) for inter-dataset generalization.

Main Results:

• Models rarely learned generally applicable representations of attack classes.
• Performance varied significantly across datasets, contradicting baseline results.
• Network-centric attacks showed better generalization (brute force, DoS, DDoS) with <5% performance loss.
• Botnets, web attacks, and infiltration models exhibited substantial performance degradation.

Conclusions:

• High classification metrics on one dataset do not guarantee generalized performance.
• Current validation strategies likely overestimate model effectiveness.
• Further research is needed to develop robust, generalizable intrusion detection models and validation methods.