Comparing Large Language Models as Health Literacy Tools: Evaluating and Simplifying Texts on gender-Affirming...

Area of Science:

• Medical Informatics
• Patient Education
• Natural Language Processing

Background:

• Patient-facing materials for gender-affirming surgery often exceed the recommended eighth-grade reading level.
• Accessible health information is crucial for informed patient decision-making.

Purpose of the Study:

• To evaluate and compare the readability analysis and text simplification capabilities of various large language models (LLMs) for online patient resources on gender-affirming procedures.
• To assess the effectiveness of LLMs in reducing the reading level of these materials to the recommended eighth-grade level.

Main Methods:

• Searched Google Incognito for 15 gender-affirming surgery terms.
• Analyzed the first 20 text results for readability using the Readability Scoring System v2.0 (RSS).
• Utilized eight LLMs to assess and simplify texts to an eighth-grade reading level, with subsequent reevaluation by RSS.

Main Results:

• Online resources averaged a reading grade level of 12.66.
• Google Gemini achieved the lowest simplified reading level (8.39), followed by Anthropic Claude (9.53) and ChatGPT 4 (10.19).
• LLMs exhibited increased error rates for feminizing/facial procedures and simplifying genital procedures.

Conclusions:

• Online patient materials for gender-affirming procedures are significantly more complex than recommended.
• While LLMs show potential for improving readability, their performance varies, necessitating cautious clinical application.
• Further research is needed to address LLM variability and bias in gender-affirming care patient education.