Use of Commercially Available Large Language Models to Generate Information Leaflets on Post-Intensive Care Syndrome:...

Area of Science:

• Artificial Intelligence in Healthcare
• Medical Informatics
• Natural Language Processing

Background:

• Patient comprehension of medical information is often limited.
• Large language models (LLMs) offer potential for simplifying complex medical content.
• LLMs can reduce the burden on healthcare providers for patient explanations.

Purpose of the Study:

• Evaluate the quality of patient information leaflets generated by commercial LLMs.
• Assess the effectiveness of different prompt designs for LLM-generated medical texts.
• Compare human and LLM-based evaluations of generated medical content.

Main Methods:

• Generated 72 informational texts on post-intensive care syndrome using 6 LLMs and 4 prompt designs.
• Provided clinical practice guidelines as reference context for LLMs.
• Conducted human evaluation of 9 selected texts by medical specialists and nonmedical personnel.
• Performed a parallel LLM-based assessment and compared scores.

Main Results:

• Generated texts received average human evaluation scores of 6.8+, with no harmful content identified.
• LLaMA 3 70B with text-augmented prompting and clinical guidelines achieved the highest score.
• Text-augmented prompting generally yielded higher scores than simple prompts.
• Evaluator ratings differed between healthcare professionals and nonprofessionals.

Conclusions:

• Commercially available LLMs can generate acceptable patient-facing informational materials.
• LLM-generated content showed potential for supporting patient education under feasibility constraints.
• Further validation with larger, diverse samples is recommended to confirm LLM performance.