AI-Driven Large Language Models in Health Consultations for HIV Patients

Area of Science:

• Artificial Intelligence in Healthcare
• Medical Informatics
• HIV Medicine

Background:

• Large language models (LLMs) are increasingly explored for healthcare applications.
• Assessing LLM performance in specialized medical consultations, such as for HIV patients, is crucial for safe clinical deployment.
• Current LLM capabilities in providing comprehensive, accurate, and empathetic health information require rigorous evaluation.

Purpose of the Study:

• To comprehensively assess the performance of advanced LLMs in health consultations for individuals living with HIV.
• To evaluate LLM applicability across multiple dimensions including medical accuracy, comprehensiveness, understandability, reliability, and humanistic care.
• To provide evidence-based insights for the clinical deployment of LLMs in HIV management.

Main Methods:

• A 23-question HIV-specific question bank was developed covering fundamental knowledge, diagnosis, treatment, prognosis, and case analysis.
• Four advanced LLMs (ChatGPT-4o, Copilot, Gemini, Claude) were evaluated using a multi-dimensional system.
• Expert scoring on a five-point Likert scale, followed by statistical analysis, consistency checks, and post-hoc testing.

Main Results:

• Claude demonstrated superior performance in comprehensiveness (4.333), understandability (3.797), and humanistic care (2.855).
• Copilot excelled in diagnostic questions (3.880), while Gemini showed exceptional performance in case analysis (4.111).
• Claude outperformed other models in thoroughness and humanistic care (P < 0.05); Gemini significantly outperformed ChatGPT in case analysis (P < 0.001). Humanistic care was a consistent weakness across all models.

Conclusions:

• Diverse LLMs exhibit task-specific strengths, indicating significant potential for HIV patient management.
• Improvements are needed in the humanistic care aspects of LLMs to enhance their efficacy in patient consultations.
• LLMs can support HIV care, but their limitations in empathetic communication require further development before widespread clinical adoption.