评估通过大型语言模型对疾病流行病学信息的响应的准确性
Kexin Zhu1, Jiajie Zhang2, Anton Klishin3
1Epidemiology and Benefit Risk, Sanofi, Bridgewater, New Jersey, USA.
Pharmacoepidemiology and drug safety
|February 4, 2025
概括
大型语言模型 (LLM) 显示出疾病流行病学研究的潜力,但当前版本存在局限性. 与Bard和ChatGPT-3.5相比,ChatGPT-4在疾病频率查询中表现出更高的准确性和一致性,尽管所有模型都产生了不准确性.
科学领域:
- 医疗信息学 医疗信息学
- 医疗保健中的人工智能
- 流行病学 流行病学
背景情况:
- 准确的疾病流行病学数据对于药物流行病学研究至关重要.
- 大型语言模型 (LLM) 越来越多地被探索其在获取和合成健康信息方面的潜力.
研究的目的:
- 评估著名的LLM (ChatGPT-3.5,ChatGPT-4,Google Bard) 在提供准确的疾病频率数据方面的表现.
- 评估疾病流行病学LLM响应的准确性,一致性和参考质量.
主要方法:
- 对三个LLM提出了有关疾病流行和发病率的21个问题.
- 从黄金标准引用的基准数据与LLM响应进行了比较.
- 系统评估了准确性,一致性 (跨不同查询日期) 和参考有效性 (相关性,真实性).
主要成果:
- 在疾病流行病学查询中,ChatGPT-4获得了最高的准确性 (76.2%) 和一致性 (71.4%).
- 谷歌巴德和ChatGPT-3.5的准确性较低 (分别为50.0%和45.2%) 和一致性较低.
- 所有的LLM都产生了不准确的响应,包括不相关或伪造的引用,ChatGPT-3.5没有提供引用,Bard具有高比例的不存在/伪造的引用.
结论:
- 在检索准确的疾病流行病学信息方面,ChatGPT-4显著优于Bard和ChatGPT-3.5.
- 尽管取得了进展,但当前的LLM存在严重的局限性,包括不准确和不可靠的参考,阻碍其在制药,学术或监管研究环境中的直接应用.
相关概念视频
Steps in Outbreak Investigation
102
In the ever-evolving field of public health, statistical analysis serves as a cornerstone for understanding and managing disease outbreaks. By leveraging various statistical tools, health professionals can predict potential outbreaks, analyze ongoing situations, and devise effective responses to mitigate impact. For that to happen, there are a few possible stages of the analysis:
102
Sensitivity, Specificity, and Predicted Value
171
In healthcare diagnostics, laboratory tests play a crucial role in identifying and diagnosing a wide range of medical conditions. However, interpreting test results is not always straightforward. An abnormal test result does not always confirm the presence of a disease, just as a normal result does not guarantee its absence. To assess the reliability of these diagnostic tools, healthcare practitioners rely on two key statistical indicators: sensitivity and specificity.
Sensitivity is the...
Sensitivity is the...
171
Statistical Methods for Analyzing Epidemiological Data
293
Epidemiological data primarily involves information on specific populations' occurrence, distribution, and determinants of health and diseases. This data is crucial for understanding disease patterns and impacts, aiding public health decision-making and disease prevention strategies. The analysis of epidemiological data employs various statistical methods to interpret health-related data effectively. Here are some commonly used methods:
293
Leaky Scanning
5.1K
During most eukaryotic translation processes, the small 40S ribosome subunit scans an mRNA from its 5' end until it encounters the first start AUG codon. The large 60S ribosomal subunit then joins the smaller one to initiate protein synthesis. The location of the translation initiation is largely determined by the nucleotides near the start codon as there may be multiple translation initiation sites present on the mRNA. Marilyn Kozak discovered that the sequence RCCAUGG (where R...
5.1K
Statistical Software for Data Analysis and Clinical Trials
480
Statistical software is pivotal in data analysis and clinical trials by providing tools to analyze data, draw conclusions, and make predictions. These software packages range from simple data management applications to complex analytical platforms, supporting various statistical tests, models, and simulation techniques. Their significance lies in their ability to handle vast amounts of data with precision and efficiency, enabling researchers to validate hypotheses, identify trends, and make...
480
Models of Health Promotion and Illness Prevention II
1.5K
The person's health status fluctuates continually, varying from being in good health to becoming ill and returning to being healthy. To understand the concept of illness prevention, there are two models. First, the health-illness continuum model is a graphic representation of an individual's wellness. It states that a person is considered healthy in the absence of physical disease and the presence of good emotional health.
The agent-host-environment model states that disease results...
The agent-host-environment model states that disease results...
1.5K


