How Well Do Different AI Language Models Inform Patients About Radiofrequency Ablation for Varicose Veins?

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Vascular Surgery

Background:

• Growing public reliance on artificial intelligence (AI) and large language models (LLMs) for medical information.
• Uncertainty regarding the accuracy and clarity of AI-generated medical advice, specifically for procedures like radiofrequency ablation (RFA) for varicose veins.

Purpose of the Study:

• To evaluate and compare the quality of AI-generated responses from leading LLMs concerning radiofrequency ablation (RFA) for varicose veins.
• To assess the reliability, clarity, and usefulness of AI answers to common patient questions about RFA, as judged by expert vascular surgeons.

Main Methods:

• A blinded, comparative observational study involving 32 vascular surgeons.
• Standardized questions on RFA for varicose veins were posed to five top LLMs: ChatGPT-4, DeepSeek-R1, Gemini 2.0, Grok-3, and LLaMA 3.1.
• Surgeons evaluated responses based on accuracy, clarity, relevance, and depth, with statistical analysis (Friedman, Wilcoxon signed-rank tests) to determine performance.

Main Results:

• Grok-3 provided the highest quality responses in 51.6% of cases, significantly outperforming other models (p < 0.0001).
• ChatGPT-4 ranked second (23.1%), with Gemini, DeepSeek, and LLaMA showing comparable, lower performance.
• Grok-3 excelled in procedural risks and post-procedure care responses, while ChatGPT-4 was best for introductory information. A significant portion of surgeons remained hesitant to recommend AI tools.

Conclusions:

• Grok-3 and ChatGPT-4 currently offer the most reliable AI-generated patient education for radiofrequency ablation (RFA) of varicose veins.
• AI tools demonstrate potential for enhancing patient understanding and reducing physician workload, but require ongoing evaluation and cautious integration.
• This study provides a benchmark for future AI technology comparisons in medical applications.