OpenTriIDS: Confidence-Guided Open-Set Intrusion Triage for Unknown Attack Detection in IoT Devices

Open-set IoT intrusion detection must distinguish unfamiliar attacks from uncertain known traffic. OpenTriIDS implements a three-action policy: a random forest accepts high-confidence predictions, routes uncertain samples to a compact known-class prototype memory, and rejects routed samples beyond a standardized Euclidean distance boundary. At the representative S1 operating points on Edge-IIoTset and CIC-IoT-2023, the operational macro-F1 reaches 68.44% and 76.81%, with routing rates of 4.53% and 4.13%, respectively. The representative S1 analysis shows that the confidence gate defines the selected subset, while the three-prototype support rule improves macro-F1 and changes the balance between known-class retention and unknown rejection; the magnitude and direction depend on the dataset and operating point. The prototype-distance module records the matched label and distance for each routed sample. A complementary controlled protocol evaluates three seeds across multiple held-out-family settings, while a seven-method S1 comparison evaluates MSP, OpenMax, ORI open recognition, EFC, post hoc Energy OOD, single-layer Mahalanobis scoring, and OpenTriIDS at their documented operating points. OpenTriIDS achieves the highest macro-F1 among the evaluated methods in both datasets under this comparison. Together, the analyses characterize a selective open-set policy whose matched-gate macro-F1 advantage is accompanied by dataset- and holdout-dependent unknown-recall trade-offs, while distinguishing predictive performance from routing and computational cost.