HepaCopilot: A 6G-Enabled Multimodal Vision-Language Agent for Real-Time Hepatocellular Carcinoma Risk Stratification
Insights
HepaCopilot, an AI tool, improves hepatocellular carcinoma (HCC) detection by analyzing contrast-enhanced ultrasound (CEUS) videos and clinical data. This multimodal agent offers interpretable risk assessments, addressing variability in current diagnostic methods.
Area of Science:
- Medical Imaging
- Artificial Intelligence
- Oncology
Background:
- Hepatocellular carcinoma (HCC) is a leading cause of cancer mortality globally.
- Contrast-enhanced ultrasound (CEUS) is vital for HCC detection but suffers from interpretation variability due to operator dependence.
- Standardized interpretation of CEUS for HCC risk assessment remains a challenge in clinical practice.
Purpose of the Study:
- To develop and evaluate HepaCopilot, a multimodal vision-language agent for interpretable HCC risk assessment.
- To integrate temporal CEUS video sequences with clinical metadata for enhanced diagnostic accuracy.
- To overcome the limitations of subjective CEUS interpretation in HCC diagnosis.
Main Methods:
- Developed HepaCopilot, a multimodal AI agent utilizing Chain-of-Thought (CoT) for structured clinical reasoning.
- Employed a hierarchical temporal encoding architecture with cross-modal attention for feature extraction across arterial, portal venous, and delayed CEUS phases.
- Evaluated HepaCopilot on a public TCIA dataset of 120 HCC subjects with CEUS examinations.
Main Results:
- HepaCopilot demonstrated strong discrimination performance with an Area Under the Receiver Operating Characteristic curve (AUROC) of 0.94 on a test set of 18 subjects.
- The AI agent systematically extracts features from multi-phase CEUS videos and integrates them with clinical metadata.
- Performance was compared against baseline methods, showing significant potential despite a wide confidence interval due to the small test set size.
Conclusions:
- HepaCopilot offers a promising AI-driven solution to enhance the accuracy and interpretability of HCC risk assessment using CEUS.
- The multimodal approach addresses the variability inherent in operator-dependent CEUS interpretation.
- Further validation on larger datasets is warranted to confirm the robustness of HepaCopilot's performance in diverse clinical settings.
Abstract:
Hepatocellular carcinoma (HCC) is one of the leading causes of cancer-related deaths around the world. One major barrier to detecting HCC is that CEUS is heavily reliant on operator interpretation, leading to an excessive amount of variability in how the test results are interpreted across different clinical settings. We developed HepaCopilot, which is a multimodal, vision-language agent that is able to integrate a sequence of temporal CEUS videos together with clinical metadata, thus providing an interpretable risk assessment of HCC using a structured method of clinical reasoning using Chain-of-Thought (CoT). HepaCopilot's hierarchical temporal encoding architecture, along with cross-modal attention mechanisms, allows the systematic extraction of features in each of the three phases: the arterial, portal venous, and delayed phases. We performed an evaluation of HepaCopilot using a publicly available dataset from TCIA B-mode-and-CEUS-Liver comprising 120 subjects who had CEUS examinations and who had been pathologically diagnosed with HCC. Using a subject-level evaluation of a test set containing 18 subjects we show that our method achieves a discrimination performance of AUROC = 0.94 (bootstrap 95% CI: 0.82-1.00, 2,000 resamples) when compared to baseline methods, though the wide confidence interval reflects the limited test set size.
