Explainable zero-day attack detection in IoMT using transformer-based time-series modeling

Securing Internet of Things (IoT) and Internet of Medical Things (IoMT) networks necessitates intrusion detection systems (IDS) robust against severe class imbalance, zero-day attacks, and deployment constraints. Prior methodologies often rely on heavy feature engineering, stacking ensembles, or data augmentation for rebalancing; yet they remain inherently closed-set and brittle across heterogeneous datasets, thereby limiting calibrated performance in real-world environments. To overcome these limitations, we propose a paradigm shift toward protocol-aware sequence modeling and principled open-set calibration. We introduce a Protocol State-Masked, Multi-scale Transformer that directly incorporates finite-state protocol constraints into the self-attention mechanism. This core architecture is augmented by two auxiliary heads (next-state prediction and violation scoring) and an energy-based open-set module calibrated using Extreme Value Theory. To rigorously verify generalization rather than simple memorization, we train the model on a heterogeneous IoMT benchmark (CICIoMT2024) and evaluate it on a completely held-out, independent testbed (WUSTL-EHMS-2020). On CICIoMT2024, the model achieves 99.62% accuracy and 99.30% macro-F1 (binary classification), and 98.90% accuracy with 98.40% macro-F1 (19-class classification). In zero-day settings, it attains an AUROC-OOD of 0.975 with an FPR@95%TPR of 7.8%. Crucially, on the unseen WUSTL-EHMS-2020 dataset, the model demonstrates strong cross-dataset transfer, reaching ≈ 99.6% overall accuracy with a balanced TPR/TNR of ≈ 99.6%. Ablation studies confirm that both the protocol masking and the auxiliary heads significantly sharpen decision boundaries, substantially reduce false alarms, and improve minority-class balance when compared to data-only Transformer variants and conventional ensembles. Our design effectively mitigates imbalance (via multi-resolution windows, class-aware objectives, and protocol-conditioned learning) and curbs overfitting (through structural supervision and calibrated open-set thresholds), delivering a calibrated, high-fidelity detection system that generalizes across diverse datasets while remaining practical for deployment.