A Future of Self-Directed Patient Internet Research: Large Language Model-Based Tools Versus Standard Search Engines.

Area of Science:

• Artificial Intelligence in Healthcare
• Medical Informatics
• Patient Education

Background:

• Generalist large language models (LLMs) are increasingly accessible to patients.
• Patients may use LLMs as an alternative to traditional search engines for health information.
• Evaluating the quality of LLM-generated health information is crucial.

Purpose of the Study:

• To assess the suitability of publicly available LLM-based chatbots for patient education.
• To compare the responses of Google, Bard, GPT-3.5, and GPT-4 to common chronic health queries.
• To evaluate physician-assessed accuracy, comprehensiveness, and quality of LLM responses.

Main Methods:

• Five common chronic health conditions (hypertension, hyperlipidemia, diabetes, anxiety, mood disorders) were selected.
• Physician-board certified experts reviewed responses from Google, Bard, GPT-3.5, and GPT-4.
• Responses were rated on a five-point Likert scale for accuracy, comprehensiveness, and quality.
• Readability was assessed using Flesch-Kincaid Grade Levels.

Main Results:

• GPT-3.5 and GPT-4 received significantly higher ratings for comprehensiveness and quality compared to Bard and Google (p < 0.05).
• Bard and Google provided more readable responses with lower average Flesch-Kincaid Grade Levels.
• Google's search results were rated significantly lower in accuracy, comprehensiveness, and quality.

Conclusions:

• Publicly available LLM-based tools show potential for providing more accurate patient information on chronic conditions than Google search.
• LLMs may serve as a valuable alternative to traditional search engines for health-related queries.
• Further research into the clinical application of LLMs for patient education is warranted.