Performance of Multimodal Artificial Intelligence Chatbots Evaluated on Clinical Oncology Cases

Area of Science:

• Artificial Intelligence in Medicine
• Oncology Diagnostics
• Medical Chatbot Technology

Background:

• Multimodal artificial intelligence (AI) chatbots can process both image and text data, potentially enhancing diagnostic accuracy compared to unimodal (text-only) AI.
• The comparative medical accuracy of multimodal versus unimodal AI chatbots in clinical oncology remains under-investigated.

Purpose of the Study:

• To assess the accuracy of multimodal and unimodal AI chatbots in responding to clinical oncology case questions.
• To evaluate the impact of prompt engineering techniques, specifically zero-shot chain-of-thought, on chatbot performance.

Main Methods:

• A cross-sectional study evaluated 10 AI chatbots (3 multimodal, 7 unimodal) using 79 clinical oncology cases from JAMA Network Learning.
• Medical accuracy was benchmarked using multiple-choice and free-text responses.
• Free-text responses were assessed by a panel of oncology specialists.

Main Results:

• No consistent superiority in accuracy was found for multimodal chatbots over unimodal chatbots.
• Top performance varied between multiple-choice (up to 72.15%) and free-text (up to 37.97%) evaluations.
• Multimodal chatbot accuracy decreased with cases involving multiple images, and 9 out of 10 chatbots showed lower accuracy in free-text compared to multiple-choice formats.

Conclusions:

• Multimodal AI chatbots did not demonstrate superior medical accuracy compared to unimodal chatbots for clinical oncology cases in this study.
• Optimization is needed for multimodal AI to effectively integrate image data for improved oncology-specific accuracy and reliability.
• Further research is essential to enhance the clinical utility of AI in oncology.