Performance of Large Language Model-Based Chatbots in Primary Health Care Teleconsultations: A Comparison Between...

Area of Science:

• Health Informatics
• Artificial Intelligence in Medicine
• Telemedicine

Background:

• Telehealth is integral to primary healthcare in Brazil, advanced by the National Telehealth Program.
• Artificial intelligence (AI) presents a promising opportunity to augment telehealth benefits.
• This study evaluates AI's performance against human responses in teleconsultation services.

Purpose of the Study:

• To compare the performance of human and AI-generated responses in teleconsultation.
• To assess quality criteria, risk potential, authorship identification, and inquiry resolution.
• To investigate AI's utility as a support tool in telehealth.

Main Methods:

• A comparative cross-sectional study of 180 human and AI responses from January 2020 to May 2024.
• Blinded evaluation based on medical adequacy, conciseness, coherence, comprehensibility, risk, and authorship accuracy.
• Analysis covered cardiology, endocrinology, and obstetrics/gynecology (OB-GYN) using various statistical tests.

Main Results:

• AI responses significantly surpassed human responses in comprehensibility across all specialties.
• No significant differences were found in medical adequacy, conciseness, coherence, risk potential, or inquiry resolution, though AI generally scored higher.
• AI demonstrated statistically significant superiority in authorship identification accuracy within endocrinology and OB-GYN.

Conclusions:

• AI shows substantial potential as a support tool for teleconsultation services.
• AI responses exhibit high quality and accuracy, comparable to human experts.
• Further integration of AI can enhance the efficiency and effectiveness of telehealth.