Interpretable Passive Multi-Modal Sensor Fusion for Human Identification and Activity Recognition.

Area of Science:

• Computer Science
• Artificial Intelligence
• Sensor Technology

Background:

• Accurate human identification and activity recognition (HIAR) is crucial for indoor monitoring.
• Single-modality sensor systems face challenges including privacy concerns, intrusion, and high costs.
• There is a need for interpretable, passive, and multi-modal solutions for long-term HIAR.

Purpose of the Study:

• To propose and validate PRF-PIR, an interpretable, passive, multi-modal sensor fusion system for HIAR.
• To address the shortcomings of single-modality systems by integrating diverse sensor data.
• To demonstrate the system's effectiveness in a real-world academic office environment.

Main Methods:

• Development of PRF-PIR, integrating a software-defined radio (SDR) and a novel passive infrared (PIR) sensor.
• Implementation of a recurrent neural network (RNN) for HIAR, capable of handling temporal data dependencies.
• Data collection involving twelve subjects performing eleven distinct activities in an office setting.

Main Results:

• The PRF-PIR system achieved high accuracy: 0.9866 for human identification and 0.9623 for activity recognition.
• Explainable Artificial Intelligence (XAI) methodologies validated the benefits of sensor fusion over single-sensor approaches.
• The system demonstrated robustness in recognizing complex and similar activities across various subjects.

Conclusions:

• PRF-PIR offers a passive, non-intrusive, and highly accurate solution for human monitoring.
• Sensor fusion significantly enhances HIAR performance compared to single-sensor deployments.
• The proposed system is suitable for long-term, privacy-preserving monitoring in diverse indoor environments.