How AI Responds to Common Lung Cancer Questions: ChatGPT vs Google Bard.

Area of Science:

• Artificial Intelligence in Medicine
• Medical Informatics
• Radiology

Background:

• Large language models (LLMs) like ChatGPT and Google Bard offer potential benefits and challenges in healthcare.
• The increasing use of AI necessitates evaluating their performance in specialized medical domains.

Purpose of the Study:

• To compare the accuracy and consistency of ChatGPT-3.5 and Google Bard in answering non-expert questions about lung cancer prevention, screening, and radiology terminology.
• To assess AI performance against established guidelines like Lung Imaging Reporting and Data System (Lung-RADS) v2022.

Main Methods:

• Forty identical questions on lung cancer were posed to ChatGPT-3.5, Google Bard, Bing, and Google Search.
• Responses were reviewed by two radiologists for accuracy (correct, partially correct, incorrect, unanswered).
• Consistency was evaluated based on agreement between AI responses, and statistical analysis was performed using Stata.

Main Results:

• ChatGPT-3.5 achieved 70.8% correct and 11.7% partially correct answers.
• Google Bard answered 79.9% of questions, with 51.7% correct and 9.2% partially correct.
• ChatGPT-3.5 showed significantly higher accuracy (OR=1.55) and consistency (OR=6.65) compared to Google Bard.

Conclusions:

• ChatGPT-3.5 outperformed other AI tools in accuracy and consistency for lung cancer-related queries.
• No AI tool provided complete accuracy or consistency, underscoring the continued importance of expert human review in medical applications.
• Further research is needed to improve AI reliability in specialized medical fields.