Multimodal LLM-driven IoT digital healthcare platform for intelligent dysphagia dietary monitoring

Patients with dysphagia who are transitioning from clinical to domestic environments face significant life-threatening risks due to the "nutrition-texture disconnect"-the systemic gap between hospital-prescribed International Dysphagia Diet Standardisation Initiative (IDDSI) levels and actual home-prepared meals. Existing dietary monitoring tools effectively track caloric intake, yet they lack the rheological sensing capabilities required to prevent acute choking and liquid aspiration hazards. This preliminary feasibility study introduces a novel, smartphone-centric Internet of Things platform that operationalizes a multimodal large language model (MLLM) as an "intelligent soft sensor" to bridge this safety gap.

The system utilizes a three-stage cascaded pipeline-incorporating chain-of-thought reasoning and bi-directional safety logic-to perform automated dish recognition, localized nutritional retrieval, and pre-consumption IDDSI texture auditing from a single smartphone image. To evaluate performance, the platform was benchmarked against the newly established Clinical-Expert IDDSI Validation Benchmark, a curated dataset of 115 diverse meal images independently verified via physical rheological testing by certified clinical specialists. A comprehensive comparative evaluation was conducted across 14 commercial and open-source models.

Our Qwen-based pipeline achieved a total exact match accuracy of 61.74%. Crucially, the system demonstrated the highest safety profile among 14 tested commercial and open-source models, achieving the lowest total false negative rate (FNR) of 9.2%. While localized open-source models exhibited a critical failure mode regarding liquid aspiration hazards (100% FNR), our cloud-based system maintained a robust 7.14% FNR for liquids.

These empirical results indicate that the proposed platform serves as a superior, safety-first screening proxy tool for digital healthcare. By prioritizing a low hazard-miss rate over raw conversational baseline accuracy, the system successfully bridges the safety gap between clinical prescriptions and daily home-based monitoring. The framework offers an ethical, non-invasive, and highly scalable soft-sensing solution for longitudinal, unsupervised dysphagia oversight in home-care settings.