Chatbots and Diabetes: Is There Gender Bias?

Area of Science:

• Artificial Intelligence in Healthcare
• Ophthalmology
• Endocrinology

Background:

• Diabetic Retinopathy (DR) is a leading cause of vision loss in diabetic patients.
• Large Language Models (LLMs) are increasingly used for health information dissemination.
• Assessing the accuracy and safety of LLM-generated health advice is crucial.

Purpose of the Study:

• To evaluate the quality of responses from four leading LLMs to a clinical scenario concerning Diabetic Retinopathy.
• To analyze readability, clinical terminology, healthcare recommendations, and privacy considerations in LLM outputs.

Main Methods:

• Four LLMs (ChatGPT-o1, DeepSeek-v3, Gemini 2.0 Flash, Claude 3.7 Sonnet) were queried with a specific patient case of Type 2 Diabetes Mellitus and vision changes.
• Responses were analyzed using Flesch-Kincaid Grade Level scoring.
• Content analysis focused on clinical terminology, healthcare advice, and privacy.

Main Results:

• All LLMs produced content at high school to college reading levels, exceeding recommended health literacy standards.
• DeepSeek-v3 used more specialized terminology and referenced specific diabetes guidelines.
• LLM responses showed varying degrees of gender bias, with DeepSeek exhibiting more discrepancy.

Conclusions:

• LLMs show potential for patient education in diabetes management and Diabetic Retinopathy.
• Essential improvements are needed in LLM readability, gender bias reduction, and output appropriateness.
• Healthcare professionals must critically review and validate LLM-generated information before patient dissemination.