Performance of Large Language Models in Answering Osteoporosis-Related Frequently Asked Questions: A Systematic...

To systematically evaluate and compare the performance of large language models (LLMs) in answering osteoporosis-related frequently asked questions (FAQs) derived from international osteoporosis-related associations. A standardized question bank was constructed based on FAQs summarized from three international osteoporosis-related associations. Six LLMs were prompted to generate responses to all questions under uniform conditions. Two osteoporosis experts independently and blindly evaluated all responses using 5-point Likert scales for accuracy and comprehensiveness. Inter-rater reliability was assessed using Cohen's κ coefficient. Nonparametric statistical analyses, including Kruskal-Wallis tests with Dunn's post hoc comparisons and Wilcoxon signed-rank tests, were performed to compare models' performance. A total of 528 responses were generated from 88 association-based questions. Inter-rater agreement was good for both accuracy and comprehensiveness (κ = 0.668 and 0.702, respectively). Significant differences were observed among models, with Claude Sonnet 4.5 demonstrating the highest overall performance on association-based FAQs. Although most models showed solid medical knowledge and basic reasoning ability, substantial variability was observed across question types and prompt formulations. Current LLMs demonstrate promising potential as auxiliary tools for osteoporosis-related health management. However, it is necessary to optimize the strategies for integrating medical knowledge and designing prompts, so as to better improve the accessibility of its clinical application.