How to Effectively Collect and Process Network Data for Intrusion Detection?

Area of Science:

• Cybersecurity
• Network Security
• Machine Learning

Background:

• The increasing number of cyberspace security breaches necessitates advanced intrusion detection systems.
• Flow-based data, particularly NetFlow, shows promise for machine learning-based intrusion detection.
• Existing benchmark datasets often lack real-world applicable features for effective intrusion detection.

Purpose of the Study:

• To identify and investigate valuable features within the NetFlow schema for effective, real-world machine learning-based network intrusion detection.
• To address the research gap concerning practical feature sets for intrusion detection systems.
• To determine the minimal amount of labeled data required for training effective intrusion detection models.

Main Methods:

• Applied several feature selection techniques on five flow-based network intrusion detection datasets.
• Established an informative flow-based feature set by analyzing NetFlow data.
• Investigated the data requirements for training machine learning algorithms in intrusion detection.

Main Results:

• Identified a concise set of 10 features from the NetFlow schema that are highly effective for intrusion detection.
• Demonstrated that a small amount of labeled data is sufficient for training high-performing intrusion detection models.
• Validated the usability of the selected features for real-world applications.

Conclusions:

• A minimal set of 10 NetFlow features can enable effective machine learning-based network intrusion detection.
• Collecting a limited amount of end-user labeled data is sufficient for practical deployment of intrusion detection systems.
• This research bridges the gap between intrusion detection research and real-world market applications.