MRCER-DR: Multi-Representation Confidence Entropy Rejection for IoT Open-Set Intrusion Detection

The rapid growth of Internet of Things (IoT) devices has made cyberattacks increasingly covert and evolving. Traditional closed-set intrusion detection systems fail to identify unknown attacks that are unseen during training. Existing open-set methods often rely on a single rejection signal, while multi-representation approaches mainly focus on feature fusion or decision aggregation rather than exploiting complementary uncertainty responses across representations. In addition, probability-based rejection may become less reliable under severe class imbalance. To address these issues, we propose Multi-Representation Confidence Entropy Rejection with Dual-Reject (MRCER-DR), an open-set intrusion detection method for IoT with low inference overhead. It constructs four representations (Raw, Graph, 2D-Stat, and Fusion) and builds a joint rejection score, Confidence Entropy Rejection (CER), combining the maximum class probability and prediction entropy. The Multi-Representation Confidence Ensemble (MRCE) then aggregates the CER scores across representations, while Dual-Reject supplements MRCE with the Mahalanobis distance under severe class imbalance. On UNSW-NB15, MRCER-DR achieves 83.1% unknown recall and an 86.2% AUROC; on ToN-IoT, these reach 97.7% and 90.9%. The single-sample inference latency ranges from 0.04 to 1.01 ms. Without modifying the classifier training, our method balances known-class classification and unknown attack detection, providing a low-overhead approach for IoT open-set intrusion detection.