Enhancing Large Language Models With AI Agents for Chronic Gastritis Management: Comprehensive Comparative Study

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Gastroenterology

Background:

• Chronic gastritis has a high prevalence and can progress to gastric cancer without timely intervention.
• Effective management necessitates comprehensive lifestyle changes and patient self-management due to limited continuous professional follow-up.
• Large language models (LLMs) show promise for patient management but have limitations like hallucinations and scope restrictions.

Purpose of the Study:

• To explore the effectiveness of artificial intelligence (AI) agents in supporting postdiagnosis self-management for chronic gastritis patients.
• To evaluate AI agents' performance from multiple perspectives, including safety, robustness, and comparison with existing LLMs.

Main Methods:

• Developed an AI agent framework integrating LLMs with retrieval-augmented generation and a search engine tool.
• Collected real patient questions from clinical settings to test the framework's performance across various difficulty levels and scenarios.
• Analyzed safety and robustness through repeated experiments and compared performance against state-of-the-art LLMs using automated metrics and expert manual assessments.

Main Results:

• AI agents significantly outperformed LLMs in handling high-complexity questions (82.849 vs 77.825 average score) and clinical consultation tasks.
• Physician evaluations indicated high safety (4.98/5) with minimal variability in responses (mean absolute deviation of 0.0167 for embedding average score).
• AI agents demonstrated superior reliability, interpretability, and practical applicability compared to advanced medical and general-domain LLMs in chronic gastritis management.

Conclusions:

• AI agents based on LLMs possess significant application value for chronic gastritis management.
• These agents can effectively guide patients with chronic diseases, potentially reducing physician workload and improving home care quality.
• The findings support the broader adoption of AI agents in healthcare settings for chronic disease management.