Identification of Online Health Information Using Large Pretrained Language Models: Mixed Methods Study

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Health Informatics

Background:

• Online health information is prevalent but often contains inaccuracies, influencing public health decisions.
• Misleading health claims pose challenges to healthcare systems.
• Large Language Models (LLMs) show potential for identifying health misinformation, but their effectiveness is under-explored.

Purpose of the Study:

• To evaluate the performance of four mainstream LLMs in identifying online health information.
• To provide empirical evidence for the practical application of LLMs in health information verification.

Main Methods:

• Collected 2708 samples of online health information (true and false claims) via web scraping from rumor-refuting websites.
• Utilized LLM APIs for authenticity verification, with expert results as benchmarks.
• Evaluated model performance using semantic similarity, accuracy, recall, F1-score, content analysis, and credibility.

Main Results:

• All four LLMs performed well in identifying online health information.
• ChatGPT-4 achieved the highest accuracy (87.27%), followed closely by Ernie Bot (87.25%) and iFLYTEK Spark (87%).
• ChatGPT-4 provided the most reliable credibility assessments, while Ernie Bot showed the highest semantic similarity to expert texts. Misjudgments were more common in nutrition-related topics.

Conclusions:

• LLMs demonstrate potential for assisting in online health information identification but exhibit performance variations.
• Models excel at generating accessible explanations but struggle with specialized medical knowledge and emerging health issues.
• Further refinement of training methodologies is crucial to enhance LLM reliability, adaptability, and ability to handle nuanced health topics and cultural contexts.