Smarter Together: Combining Large Language Models and Small Models for Physiological Signals Visual Inspection

Area of Science:

• Artificial Intelligence in Medicine
• Machine Learning for Healthcare
• Clinical Decision Support Systems

Background:

• Large language models (LLMs) show promise in medical time-series interpretation but lack domain specificity and fine-tuning capabilities.
• Small specialized models (SSMs) excel at focused tasks but lack broad reasoning for complex medical decisions.
• Existing AI approaches face challenges in balancing generalizability with domain-specific precision and trustworthiness.

Purpose of the Study:

• To introduce Conformalized Multiple Instance Learning (ConMIL), a novel framework synergizing LLMs and SSMs for enhanced medical time-series analysis.
• To develop a Multiple Instance Learning (MIL) mechanism, QTrans-Pooling, for identifying clinically relevant physiological signal segments with per-class interpretability.
• To integrate conformal prediction with MIL for reliable, set-valued outputs and to structure these outputs for LLM enhancement.

Main Methods:

• Developed QTrans-Pooling, a novel MIL mechanism for interpretable identification of critical physiological signal segments.
• Integrated conformal prediction with MIL to provide statistically reliable, set-valued outputs, quantifying uncertainty.
• Structured interpretable and uncertainty-quantified SSM outputs to augment LLM visual inspection capabilities.

Main Results:

• ConMIL significantly enhanced the accuracy of LLMs (ChatGPT4.0, Qwen2-VL-7B, MiMo-VL-7B-RL) in arrhythmia detection and sleep stage classification.
• ConMIL-supported Qwen2-VL-7B and MiMo-VL-7B-RL achieved high precision (94.92%, 96.82%) on confident samples.
• Performance on uncertain samples improved substantially compared to LLMs used alone, demonstrating ConMIL's effectiveness in handling ambiguity.

Conclusions:

• Integrating interpretable and uncertainty-quantified SSMs with LLMs offers a promising pathway for trustworthy AI-driven clinical decision support.
• ConMIL demonstrates a novel approach to leverage the strengths of both LLMs and SSMs for complex medical data interpretation.
• The framework enhances AI interpretability and reliability, crucial for adoption in clinical settings.