Evaluation of Confusion Behaviors in SEI Models

Area of Science:

• Radio Frequency Machine Learning (RFML)
• Signal Processing
• Machine Learning

Background:

• Specific Emitter Identification (SEI) uses RFML for signal classification.
• Existing research focuses on successful model architectures for SEI.
• Systemic failures and learned behaviors in RFML for SEI are under-examined.

Purpose of the Study:

• Investigate failure patterns in RFML-based SEI models.
• Analyze classification errors across various model architectures.
• Examine the impact of Signal-to-Noise Ratio (SNR) and training data quantity on SEI performance.

Main Methods:

• Evaluated multiple RFML model architectures on a 64-radio SEI dataset.
• Controlled for Signal-to-Noise Ratio (SNR) and training data volume.
• Isolated common patterns in misclassification results.

Main Results:

• RFML models frequently defaulted to a small subset of classes (approx. 10%) as SNR decreased.
• Error patterns were consistent across different SEI models and architectures.
• Ensemble models demonstrated greater resilience at low SNR but were not optimal at high SNR.

Conclusions:

• Understanding RFML failure modes is crucial for robust SEI systems.
• Model brittleness increases with decreasing SNR.
• Ensemble methods offer a trade-off between robustness and peak performance in SEI.