BigFlow-NIDS: A large-scale dataset for network intrusion detection in big data environment.

Area of Science:

• Computer Science
• Cybersecurity
• Data Science

Background:

• Network intrusion detection systems (NIDS) are crucial for cybersecurity.
• Big data environments present challenges for traditional NIDS research.
• Existing datasets may lack the scale and detail required for modern threats.

Purpose of the Study:

• Introduce BigFlow-NIDS, a large-scale NetFlow-based dataset.
• Facilitate research in network intrusion detection within big data environments.
• Support the development of scalable, temporally-aware NIDS.

Main Methods:

• Collected and curated a dataset of 66,935,021 network flows.
• Included 55 flow attributes and 32 fine-grained attack categories.
• Formatted data in both CSV and Parquet for diverse analytical needs.

Main Results:

• Parquet format demonstrated significantly faster data loading compared to CSV (27.35s vs 920.82s).
• Dataset comprises 36.6 million benign flows and 30.3 million attack flows, highlighting class imbalance.
• Baseline analyses and anomaly detection experiments were performed.

Conclusions:

• BigFlow-NIDS is a valuable resource for big data NIDS research.
• Columnar storage formats like Parquet are essential for handling large NIDS datasets efficiently.
• The dataset enables the evaluation of scalable and temporally-aware intrusion detection systems.