Interactive multimodal disease risk evaluation using a self-service non-invasive device

Non-communicable diseases such as diabetes, fatty liver disease and chronic kidney disease are major global health burdens that benefit from early-stage detection. However, standard diagnostic methods are often time-consuming and dependent on clinical infrastructure, limiting their applicability in routine or population-scale screening.

We develop an interactive multimodal disease risk evaluation system based on a self-service, non-invasive device that captures facial images, tongue images and pulse signals. A representation learning framework was employed to calculate cross-modal dependencies for disease prediction. Analyses were performed on a real-world cohort of 2,423 participants spanning nine common non-communicable diseases. Model performance was evaluated using 10 repeated train-test splits, and paired statistical tests were conducted to assess the significance of performance differences.

Here we show that the system achieves a mean diagnostic accuracy of 92.64% across 10 repeated train-test splits. The interactive multimodal approach improves average performance by 3.39% compared with single-modality baselines, indicating that modeling cross-modal interactions enhances prediction. The importance of cross-modal patterns is analyzed, demonstrating that modalities contribute complementary diagnostic signals to improve the model's capability.

Our findings demonstrate that combining non-invasive sensing with interactive multimodal framework supports accessible and scalable disease risk evaluation, and establish iMES as a deployable solution for decentralized disease prevention in community and low-resource settings.