Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study.

Area of Science:

• Artificial Intelligence in Medicine
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) are increasingly used for medical information generation.
• Rigorous assessment of AI-generated medical content quality, accuracy, and safety is crucial across specialties.
• The rapid advancement of AI necessitates evaluating its role in healthcare.

Purpose of the Study:

• To evaluate the medical content generation performance of four prominent LLMs: Claude-instant-v1.0, GPT-3.5-Turbo, Command-xlarge-nightly, and Bloomz.
• To assess AI-generated therapeutic recommendations in ophthalmology, orthopedics, and dermatology.
• To compare physician evaluations with automated GPT-4 assessments for LLM-generated medical content.

Main Methods:

• Physician evaluation of AI-generated therapeutic recommendations for 60 diseases using mDISCERN scores, correctness, and harmfulness.
• Statistical analysis (ANOVA, t-tests) to compare model and specialty performance.
• Automated evaluation using GPT-4, compared to physician assessments via Pearson correlation.

Main Results:

• Claude-instant-v1.0 achieved the highest mDISCERN score; Bloomz had the lowest.
• Significant differences in content quality and safety were observed across LLMs and specialties.
• GPT-3.5-Turbo demonstrated the lowest harmfulness rating; common errors included diagnostic confusion and omitted treatments.
• GPT-4 evaluations showed substantial alignment with physician assessments.

Conclusions:

• LLMs demonstrate capability in generating medical content, but quality and safety require refinement.
• Regular, methodical assessments and oversight are essential for trustworthy AI medical advice.
• GPT-4 offers a scalable method for automated, domain-agnostic evaluation of AI-generated medical content.