Evaluation of the reliability, quality, and readability of multiple large language model responses (ChatGPT, Gemini,...

BackgroundInadequate or misleading health information about silicosis can delay diagnosis, reporting, and exposure removal, potentially affecting risk communication and timely occupational health interventions.ObjectiveThis study evaluates the reliability, quality, and readability of Large Language Models (LLMs) including ChatGPT (GPT-5), Gemini (2.5 Flash), and Copilot (Smart GPT-5), in answering patient-oriented silicosis questions, focusing on risk communication rather than clinical decisions.MethodsSilicosis-related questions were collected from online forums (September 1-7, 2025), and 25 frequently asked, clinically relevant questions were selected and submitted to the LLMs (September 8-14, 2025). Two occupational medicine specialists/pulmonologists independently evaluated responses (September 15-30, 2025) for reliability, quality, and readability using modified DISCERN, Global Quality Scale, and readability indices. Analyses used SPSS version 31.0.ResultsThe LLM-generated responses demonstrated fair reliability. Median (min-max) scores were 23 (12-27) for ChatGPT, 21.5 (14-29) for Gemini, and 26 (17.5-31) for Copilot. Copilot scored significantly higher than ChatGPT and Gemini (p = 0.014). GQS scores were similar across models (p = 0.102), with median scores of 4 (3-5) for ChatGPT and Copilot and 4 (2-5) for Gemini. Readability levels corresponded predominantly to secondary or tertiary education, with no significant differences between the models (all p > 0.05).ConclusionLLMs may enhance worker awareness but have limitations in accuracy and readability. They should complement, not replace, expert-supervised occupational health services. Future studies should align LLMs with international standards, improve accessibility, and establish governance frameworks for transparent and equitable AI use in occupational health.