Intrusion Detection Datasets for IIoT and ICS: A Taxonomic Review with a Decision-Aid Scoring Rubric

Area of Science:

• Cybersecurity
• Machine Learning
• Industrial Control Systems (ICS)

Background:

• Dataset quality is critical for machine learning (ML) models in intrusion detection systems (IDS) for cyber-physical industrial control systems (CPS/ICS) and the Industrial Internet of Things (IIoT).
• Existing surveys offer limited qualitative comparisons of ICS/OT/IIoT datasets, hindering effective selection for ML-based IDS.
• There is a need for quantitative documentation and decision-aid tools to evaluate these datasets.

Purpose of the Study:

• To quantitatively analyze and compare 23 ICS/OT/IIoT datasets based on measurable axes.
• To introduce a checklist for documentation completeness and a decision-aid rubric for dataset evaluation.
• To identify structural gaps in current datasets regarding multi-stage adversary behavior and protocol representation.

Main Methods:

• Analysis of 23 ICS/OT/IIoT datasets along seven measurable dimensions.
• Mapping of dataset attacks to MITRE ATT&CK for ICS tactics.
• Development and application of a documentation completeness checklist (0-7) and a decision-aid rubric (0-15) assessing realism, attack diversity, class imbalance, documentation, and reproducibility.

Main Results:

• 60.9% (14 of 23) of datasets are built on physical testbeds; 95.7% (22 of 23) map to MITRE ATT&CK for ICS, covering 11 of 12 tactics.
• Protocol coverage is heavily skewed towards Modbus (57%), with underrepresentation of protocols like Profinet and OPC UA.
• Significant structural gaps exist in capturing multi-stage adversary behavior across the analyzed datasets.

Conclusions:

• Current ICS/OT/IIoT datasets have limitations in realism, attack diversity, documentation, and protocol representation for ML-based IDS.
• A quantitative framework including a checklist and rubric aids in objective dataset evaluation.
• Dataset selection should prioritize realism and reproducibility, alongside consideration for protocol diversity and advanced persistent threat (APT) representation.