Multi-modal Cardio-respiratory Monitoring of Preterm Infants in NICU Using RGB Camera and mmWave Radar

Recent advances of contactless camera or radar based Heart Rate (HR), Respiratory Rate (RR), and Heart Rate Variability (HRV) monitoring in Neonatal Intensive Care Units (NICUs) have attracted interest. We collected data from 50 preterm infants in NICU to investigate the performance of camera and radar under realistic challenges in NICU (e.g., incubator glass, shading cloth), as well as their fusion for multi-modal contactless monitoring. Results show that the incubator glass occlusion significantly impacts radar-based HR monitoring (Mean Absolute Error (MAE) increased by 5.69 bpm; Pearson's R-value decreased by 0.26, p$< $0.01), while shading cloth mainly affects camera-based HR monitoring (MAE increased by 3.88 bpm; Pearson's R-value decreased by 0.19, p$< $0.01). Both occlusions have negative impact on HR monitoring of both modalities but minimally influence RR monitoring. The findings indicate that camera-radar fusion offers superior performance over a single modality, demonstrating the complementary benefits in solving occlusion challenges caused by incubator glass or shading cloth in NICU. We introduced four Signal Quality Indices (SQIs) for reliable signal selection and proposed a Poly-parameter Attention (PPA) model to optimize multi-SQIs fusion weights. The PPA model outperforms single-modal, single-SQI and average-weighted approaches by achieving MAE$\leq$5 bpm for both HR and RR monitoring in NICU.