Multimodal LLM for Patient Activity Recognition: Integrating Video, Audio, and Text in Clinical Environments

Area of Science:

• Artificial Intelligence
• Clinical Informatics
• Healthcare Technology

Background:

• Accurate patient activity recognition is crucial for hospital safety and care quality.
• Current methods face challenges with complex activities and clinical variability.
• Need for enhanced, patient-specific monitoring in healthcare settings.

Purpose of the Study:

• To introduce ClinActNet, a novel multimodal framework for accurate patient activity recognition.
• To improve patient monitoring by integrating diverse data sources and patient-specific context.
• To enhance clinical decision-making through interpretable AI in healthcare.

Main Methods:

• Developed ClinActNet, a multimodal framework using large language models and a context-aware Consultation Transformer.
• Integrated video, audio, and clinical documentation (EHRs).
• Incorporated a patient profile encoder, Clinical Knowledge Graph Reasoner, and personalization layer for patient-specific monitoring.

Main Results:

• Achieved 89.7% accuracy and 98.2% precision for critical safety events on 672 hours of hospital data.
• Demonstrated robustness on the public VAST dataset with 84.1% accuracy.
• Reduced overlooked critical incidents by 48% in real-world implementations.

Conclusions:

• ClinActNet offers a robust and accurate solution for patient activity recognition in hospitals.
• The framework enhances patient-centered AI and interpretable insights in clinical environments.
• ClinActNet has the potential to significantly improve patient safety and care quality.