ChatRadio-Valuer: A Chat Large Language Model for Generalizable Radiology Impression Generation on Multi-Institution...

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Natural Language Processing in Healthcare

Background:

• Conventional AI models face challenges in achieving clinical-level performance for diverse radiology tasks.
• Advancements in general AI and accessibility of radiology reports create opportunities for new diagnostic tools.

Purpose of the Study:

• To introduce ChatRadio-Valuer, a novel, general radiology diagnosis large language model (LLM) designed for localized hospital deployment.
• To evaluate the clinical-level performance of ChatRadio-Valuer across multiple institutions and disease systems.

Main Methods:

• Rigorous, full-spectrum assessment including engineering metrics, clinical validation, and efficiency evaluation.
• Benchmarking against OpenAI's GPT-3.5 and GPT-4, and comparing performance to average radiology experts.
• Utilizing a large dataset of clinical-level events (n=332,673) across five human systems and six institutions.

Main Results:

• ChatRadio-Valuer achieved 15 state-of-the-art results, surpassing GPT-3.5 and GPT-4 in comprehensive disease diagnosis.
• Demonstrated superior performance compared to the average level of radiology experts.
• Showcased zero-shot transfer learning capabilities, enhancing its utility as a radiology assistant.

Conclusions:

• Localized LLMs represent a crucial development for hospital AI applications.
• ChatRadio-Valuer demonstrates the potential for widespread clinical use in multi-institution and multi-system radiology diagnosis.
• The model ensures privacy adherence and scalability for large patient populations.