Finding Explanations in AI Fusion of Electro-Optical/Passive Radio-Frequency Data.

Area of Science:

• Artificial Intelligence
• Sensor Fusion
• Data Science

Background:

• The Information Age relies heavily on blackbox algorithms, obscuring data usage.
• Sensor fusion is crucial for enhancing model robustness and performance.
• Understanding data contributions in sensor fusion is challenging.

Purpose of the Study:

• To demonstrate an explainable Long Short-Term Memory (LSTM-CCA) model for fusing Passive RF (P-RF) and Electro-Optical (EO) data.
• To gain insights into the utilization of P-RF data within a fused sensor model.
• To improve object detection and tracking through explainable sensor fusion.

Main Methods:

• Utilized a Long Short-Term Memory (LSTM-CCA) model for sensor fusion.
• Processed P-RF data from in-phase and quadrature (I/Q) components via histograms.
• Combined P-RF data with enhanced EO data using dense optical flow (DOF).
• Employed a greedy algorithm (explainX.ai) to determine input data impact.

Main Results:

• Achieved object detection and tracking using the fused P-RF and EO data.
• The explainable LSTM-CCA framework provided insights into canonical variate contributions.
• Demonstrated scenario-specific weighting of input data for the fusion model.

Conclusions:

• Introduced a novel explainable LSTM-CCA framework for P-RF and EO sensor fusion.
• The framework offers insights into the sensor fusion process for target detection and differentiation.
• Enables decision-makers to understand and assign weights to individual sensor inputs.