Deep multi-modal features based spatio-temporal video regression for non-invasive hemoglobin estimation

The hemoglobin concentration in blood is vital for diagnosing anemia and monitoring the various health conditions. However, conventional measurement methods need invasive blood sampling so that they might have limited accessibility and uncomfortable for patients. Today, non-invasive alternatives powered by machine learning techniques provide promising solutions for point-of-care facilities and remote healthcare systems. This paper presents a methodology through a comprehensive research and development process to estimate hemoglobin levels from facial videos using multi-modal feature extraction and ensemble learning techniques. A dataset of 260 participants with various blood hemoglobin levels was processed to extract the features from pre-trained convolutional neural-networks (MobileNetV2, ResNet152), remote photoplethysmography (rPPG) signals, and color statistical features. Using these features, hemoglobin concentration was estimated via a number of machine learning models including XGBoost, Random Forest, and Stacking Regressor, respectively. Stacking Regressor provided the best estimation scores with a mean-absolute error of 0.7754 g/dL, Pearson correlation-coefficient of 0.7878, and [Formula: see text] score of 0.5852. ResNet152 model based features were combined with XGBoost, which achieved comparable performance (MAE: 0.6635 g/dL, [Formula: see text]: 0.4977). Experimental results demonstrated that multi-modal feature strategy outperformed single-modality approaches in terms of prediction accuracy and robustness. The proposed video-based estimation of hemoglobin concentration system achieves clinically relevant accuracy levels, outperforms to literature methods, comparable to point-of-care instruments demonstrating strong potential for use in anemia screening and remote patient monitoring.