Related Experiment Video
Updated: May 14, 2025
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
A clinician-based comparative study of large language models in answering medical questions: the case of asthma
Yong Yin1,2,3,4, Mei Zeng2, Hansong Wang5
1Department of Respiratory Medicine, Hainan Branch, Shanghai Children's Medical Center, School of Medicine, Shanghai Jiao Tong University, Sanya, China.
Objective:
This study aims to evaluate and compare the performance of four major large language models (GPT-3.5, GPT-4.0, YouChat, and Perplexity) in answering 32 common asthma-related questions.
Materials And Methods:
Seventy-five clinicians from various tertiary hospitals participated in this study. Each clinician was tasked with evaluating the responses generated by the four large language models (LLMs) to 32 common clinical questions related to pediatric asthma. Based on predefined criteria, participants subjectively assessed the accuracy, correctness, completeness, and practicality of the LLMs' answers. The participants provided precise scores to determine the performance of each language model in answering pediatric asthma-related questions.
Results:
GPT-4.0 performed the best across all dimensions, while YouChat performed the worst in all dimensions. Both GPT-3.5 and GPT-4.0 outperformed the other two models, but there was no significant difference in performance between GPT-3.5 and GPT-4.0 or between YouChat and Perplexity.
Conclusion:
GPT and other large language models can answer medical questions with a certain degree of completeness and accuracy. However, clinical physicians should critically assess internet information, distinguishing between true and false data, and should not blindly accept the outputs of these models. With advancements in key technologies, LLMs may one day become a safe option for doctors seeking information.
Related Concept Videos
01:30Asthma-IV: Diagnostic and Management
Clinical Assessment for Asthma:
This is the first step in diagnosing and managing asthma. It includes:
01:27Chronic Obstructive Pulmonary Disease-IV: Assessement and Diagnostic Studies
Medical History
01:29Asthma-I: Introduction
01:26Asthma-II: Pathophysiology and Classification
Additionally, environmental and genetic factors play crucial roles in determining an individual's susceptibility to asthma and the severity of their condition.
Critical processes in asthma pathophysiology include:
01:20Asthma: Pathogenesis and Management
Asthma is classified as allergic and non-allergic. Allergens such as dust mites, pollen, and pet dander trigger allergic asthma, while factors like cold air, intense emotions, or exercise can induce non-allergic asthma.
01:25Antiasthma Drugs: Mast Cell Stabilizers and Anti-IgE Drugs
Mast cell stabilizers, such as cromolyn (also known as sodium cromoglycate) and nedocromil (Tilade), are effective drugs in asthma management. These stabilizers hinder histamine release by skillfully obstructing the activation of mast cells and other cellular entities. Notably, they navigate this task without...