Interactive computer-aided diagnosis on medical image using large language models.

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Natural Language Processing

Background:

• Computer-aided diagnosis (CAD) systems have advanced medical image analysis.
• Large language models (LLMs) show promise in clinical applications but struggle with medical image interpretation.
• Integrating LLMs with CAD is crucial for enhancing diagnostic capabilities.

Purpose of the Study:

• To develop a novel strategy integrating LLMs with CAD networks for improved medical image analysis.
• To leverage LLMs' medical knowledge and reasoning to enhance CAD outputs like diagnosis, segmentation, and report generation.
• To create higher-quality, patient-friendly reports and improve the performance of vision-based CAD models.

Main Methods:

• Developed a framework integrating LLMs with existing CAD networks.
• Utilized LLMs to summarize medical image information and enhance CAD outputs.
• Employed ChatGPT and GPT-3 for natural language summarization and report generation in chest X-ray analysis.

Main Results:

• The integrated LLM-CAD framework significantly improved diagnosis performance.
• ChatGPT integration led to a 16.42 percentage point improvement in diagnosis performance for chest X-rays.
• GPT-3 integration resulted in a 15.00 percentage point F1-score improvement.
• Generated reports were of higher quality and enhanced the performance of vision-based CAD models.

Conclusions:

• The proposed strategy effectively integrates LLMs with CAD for superior medical image analysis.
• This approach enhances diagnostic accuracy, enables accurate report generation, and improves patient communication.
• The framework has the potential to revolutionize clinical decision-making and patient interaction in healthcare settings.