Cardiology-Chat: A Multi-LLMs Powered System for Cardiac Diagnostic Reasoning and Clinical Support

Area of Science:

• Cardiology
• Artificial Intelligence
• Natural Language Processing

Background:

• Cardiovascular diseases are a major cause of death globally.
• Accurate diagnosis of cardiovascular diseases is challenging.
• Existing Large Language Models (LLMs) face limitations like hallucination and poor domain-specific reasoning in cardiology.

Purpose of the Study:

• To develop an LLM-based system, Cardiology-Chat, to overcome the limitations of current LLMs in cardiovascular disease diagnosis.
• To enhance diagnostic accuracy and reliability in cardiology.

Main Methods:

• Developed Cardiology-Chat, an LLM system with a three-step reasoning framework: query parsing (Llama 3.1 8B-instruct), evidence retrieval (Retrieval-Augmented Generation - RAG) from a specialized cardiovascular knowledge base, and diagnostic conclusion generation (fine-tuned Llama model).
• Created a specialized cardiovascular vector knowledge base from diverse data sources.
• Developed a Chain-of-Thought-augmented dataset to improve LLM reasoning.
• Utilized multiple LLMs to mitigate self-consistency bias.

Main Results:

• Cardiology-Chat demonstrated significant performance improvements in experiments.
• Achieved an accuracy of 0.796 on public cardiology QA datasets.
• Attained an F1 score of 0.807 on real clinical cases.

Conclusions:

• Cardiology-Chat effectively addresses key limitations of LLMs in cardiology, including hallucination and inadequate reasoning.
• The system shows strong potential for improving the accuracy and reliability of cardiovascular disease diagnosis.
• The specialized knowledge base and reasoning augmentation techniques are crucial for the system's success.