Quality and Readability of Large Language Models' Responses to Oral Lichen Planus Patients' FAQs

Area of Science:

• Artificial Intelligence in Medicine
• Oral Medicine
• Medical Informatics

Background:

• Oral lichen planus (OLP) is a chronic inflammatory condition affecting oral mucosa.
• Patient education is crucial for managing OLP, but access to reliable information can be challenging.
• Large language models (LLMs) are emerging as potential tools for information dissemination.

Purpose of the Study:

• To assess the quality and readability of LLM-generated responses to frequently asked questions (FAQs) about OLP.
• To compare the performance of different LLMs in providing OLP information to patients.

Main Methods:

• Three LLMs (ChatGPT-4o, Gemini 2.0 Flash Experimental, Copilot) were queried with 13 patient-centered OLP FAQs.
• Responses were evaluated by 14 oral medicine experts using the Quality Assessment of Medical Artificial Intelligence (QAMAI) tool.
• Readability was assessed using Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKG) metrics.

Main Results:

• LLM responses generally demonstrated good to very good quality (median QAMAI scores).
• ChatGPT provided slightly more complete answers, especially regarding OLP definition and treatment.
• Reference provision was inconsistent; readability analysis indicated a need for college-level literacy, with varying complexity among LLMs.

Conclusions:

• LLMs show promise as supplementary tools for OLP patient education.
• Limitations include incomplete information, inconsistent referencing, and suboptimal readability, necessitating further development.
• Future research should focus on longitudinal evaluations and model training for improved accuracy and accessibility tailored to user literacy levels.