How Does ChatGPT Use Source Information Compared With Google? A Text Network Analysis of Online Health Information

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing for Health Information Retrieval
• Orthopaedic Surgery Information Access

Background:

• The public increasingly uses large language models (LLMs) like ChatGPT for medical information.
• ChatGPT provides information without citing sources, raising concerns about reliability.
• Text network analysis can infer ChatGPT's sources and compare them to traditional search engines.

Purpose of the Study:

• To compare content similarity between ChatGPT and Google Search for orthopaedic surgery queries.
• To analyze how medical consensus affects source distribution and text similarity.
• To evaluate differences in text similarity between ChatGPT 3.5 and 4.0.

Main Methods:

• Three orthopaedic queries with varying medical consensus levels were used.
• Text network analysis with term frequency-inverse document frequency (TF-IDF) compared ChatGPT and Google Search responses.
• ChatGPT 3.5 and 4.0 responses were analyzed for text similarity and source category dominance.

Main Results:

• ChatGPT responses showed significant text similarity to top Google Search results.
• Source similarity between ChatGPT and Google Search reflected the dominant source category for each topic.
• ChatGPT 4.0 demonstrated higher text similarity to Google Search results than ChatGPT 3.5.

Conclusions:

• ChatGPT's orthopaedic surgery information is similar to individual Google Search results, but source quality varies with medical consensus.
• For low-consensus topics, ChatGPT may rely on less reliable, non-academic sources.
• Physicians should guide patients on using AI tools, especially when medical evidence is limited.