AI-driven Medical Care: Evaluation of Large Language Models in Generating Personalized Stroke Education Materials

Large language models (LLMs) demonstrate remarkable potential in healthcare communication. However, whether they can process complex, high-volume medical information, such as stroke-related content, remains insufficiently validated. This study aimed to evaluate the natural language processing capabilities of LLMs in handling such content and to develop an evaluation instrument.

A survey compared educational materials generated by two LLMs (ChatGPT 4.0 and Claude 3) with neurologist-authored content on stroke. The materials were based on two clinical scenarios representing distinct stroke etiologies: cardioembolism and large-artery atherosclerosis. They were evaluated in terms of accuracy, legality, ethics, comprehensiveness, and information delivery. Scores for comprehensiveness and information delivery were compared according to participants' agreement with the use of LLMs in healthcare.

ChatGPT received the highest scores across all domains, except for legality in Scenario 2. In Scenario 1, the ranking for accuracy and summarization of clinical information was, from highest to lowest, ChatGPT, Claude, and the neurologist (η2 = 0.140, p < 0.001; η2 = 0.175, p < 0.001). The same hierarchy was observed in Scenario 2 for accuracy (η2 = 0.077, p < 0.001) and summarization (η2 = 0.194, p < 0.001). Participants who agreed with the use of LLMs in healthcare assigned higher scores for the comprehensiveness (Scenario 1, p = 0.005; Scenario 2, p = 0.007) and information delivery (Scenario 1, p = 0.003; Scenario 2, p = 0.026) of ChatGPT-generated materials than participants who did not agree.

LLMs demonstrated adequate capability to convey complex content, such as stroke-related information, in an accessible and understandable manner for non-experts.