Related Experiment Video
Updated: Jun 27, 2025
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
548
Evaluating ChatGPT's Utility in Medicine Guidelines Through Web Search Analysis
Jeremy A Dubin1, Sandeep S Bains1, Daniel Hameed1
1LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, MD, USA.
The Permanente Journal
|April 26, 2024
Summary
ChatGPT and Google Search provide different answers to health questions. Patients and doctors should carefully evaluate AI-generated health information and consult professionals for accurate medical guidance.
Area of Science:
- Medical Informatics
- Artificial Intelligence in Healthcare
Background:
- Machine learning (ML) is increasingly used in healthcare, impacting prognostic models and pattern detection.
- Chat Generative Pre-trained Transformer (ChatGPT) is an advanced ML model generating human-like text.
- Evaluating AI tools like ChatGPT for patient inquiries is crucial.
Purpose of the Study:
- To compare the capabilities of ChatGPT and Google Search in addressing patient health inquiries.
- To analyze the nature and sources of questions and answers provided by both platforms.
- To assess the reliability of AI-generated health information.
Main Methods:
- Utilized 25 search terms from American Academy of Family Physicians clinical guidelines.
- Prompted ChatGPT to find top questions related to each search term.
- Compared questions and answers from ChatGPT and Google Search, noting source domains and numerical replies.
Main Results:
- 42% of questions were consistent between ChatGPT and Google Search.
- ChatGPT sourced more from academic (38%) and government (50%) domains compared to Google Search (15% academic, 39% government).
- Google Search utilized more commercial sources (32%) than ChatGPT (11%). 39% of questions had divergent answers, with 16 ChatGPT answers advising professional consultation instead of numerical data.
Conclusions:
- ChatGPT and Google Search yield different questions and answers for health queries.
- Caution is advised for patients and physicians using ChatGPT as a health advisor.
- Medical professionals should guide patients in discussing online health information and inquiries.
More Related Videos
Related Concept Videos
01:23Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
125
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
125
01:16Clinical Trials
6.7K
Clinical trials are prospective experimental studies conducted on humans to determine the safety and efficacy of treatments, drugs, diet methods, and medical devices. Using statistics in clinical trials enables researchers to derive reasonable and accurate conclusions from the collected data, allowing them to make wise decisions in uncertain situations. In medical research, statistical methods are crucial for preventing errors and bias.
There are four phases in a clinical trial. A phase one...
There are four phases in a clinical trial. A phase one...
6.7K
01:12Hazard Ratio
115
The hazard ratio (HR) is a widely used measure in clinical trials to compare the risk of events, such as death or disease recurrence, between two groups over time. It reflects the ratio of hazard rates—the instantaneous risk of the event occurring—between a treatment group and a control group. This measure provides valuable insights into the relative effectiveness of a treatment by assessing how the risk of an event differs between the two groups.
For example, in a clinical trial...
For example, in a clinical trial...
115