Multiple large language models versus clinical guidelines for postmenopausal osteoporosis: a comparative study of
Chun-Ru Lin1, Yi-Jun Chen2, Po-An Tsai3
1Department of Orthopaedic Surgery, Chang Gung Memorial Hospital, Linkou Branch, No. 5, Fuxing Street, Guishan District, Taoyuan City, 333, Taiwan.
Abstract:
The study assesses the performance of AI models in evaluating postmenopausal osteoporosis. We found that ChatGPT-4o produced the most appropriate responses, highlighting the potential of AI to enhance clinical decision-making and improve patient care in osteoporosis management.
Purpose:
The rise of artificial intelligence (AI) offers the potential for assisting clinical decisions. This study aims to assess the accuracy of various artificial intelligence models in providing recommendations for the diagnosis and treatment of postmenopausal osteoporosis.
Methods:
Using questions from the 2020 American Association of Clinical Endocrinologists (AACE) guidelines for osteoporosis, AI models including ChatGPT-3.5, ChatGPT-4.0, ChatGPT-4o, Gemini, Gemini Advanced, and Copilot were prompted. Responses were classified as accurate if they did not contradict the clinical guidelines. Two additional categories, over-conclusive and insufficient, were created to further evaluate responses. Over-conclusive was designated if AI models provided recommendations not specified in the guidelines, while insufficient indicated a failure to provide relevant information included in the guidelines. Chi-square tests were employed to compare categorical outcomes among different AI models.
Results:
A total of 42 clinical questions were evaluated. ChatGPT-4o achieved an accuracy of 88%, ChatGPT-3.5 57.1%, ChatGPT-4.0 64.3%, Gemini 45.2%, Gemini Advanced 57.1%, and Copilot 47.6% (p < 0.001).
Conclusions:
The study reveals significant response accuracy variations across each AI model, with ChatGPT-4o demonstrating the highest accuracy. Further research is necessary to explore the broader applicability of AI in the medical domains.
Related Concept Videos
01:31Osteoclasts in Bone Remodeling
01:20Study Designs in Epidemiology
Observational studies are those where the researcher does not intervene but rather observes natural variations. They include cross-sectional, cohort, and...
01:29Bone Disorders
Bone deposition is also affected by the levels of sex hormones like estrogen and testosterone that promote osteoblast activity and bone matrix synthesis. When the level of these hormones decreases due to aging, it causes a reduction in bone deposition. As a result, bone resorption by osteoclasts...
01:40Bone Remodeling
01:17Hormones and Bone Tissue
Hormones That Influence Osteoblasts and/or Maintain the Matrix
Several hormones are necessary for controlling bone growth and maintaining the bone matrix. The pituitary gland secretes growth hormone (GH), which, as its name implies, controls bone growth. This happens in several ways: first, it triggers chondrocyte...
01:25Role of Vitamins in Maintaining Bone Health
Vitamin A
Vitamin A is involved in the process of bone remodeling. Retinoic acid, the active metabolite of Vitamin A, has nuclear receptors in osteoblasts and osteoclasts, which are involved in bone remodeling.
Vitamin B12
Vitamin B12 acts as a cofactor during the formation of osteoblast-related proteins, such as osteocalcin. Vitamin B12 plays a role...