不是你正在寻找的模型:传统的ML在临床预测任务中优于LLM
Katherine E Brown1, Chao Yan1, Zhuohang Li2
1Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee.
medRxiv : the preprint server for health sciences
|December 16, 2024
概括
大型语言模型 (LLM) 在使用电子健康记录 (EHR) 的临床预测任务中显示出比传统机器学习 (ML) 更低的性能和稳定性. 虽然LLM正在改善,但本地ML模型在准确性和隐私弹性方面仍然优越.
科学领域:
- 生物医学信息学 生物医学信息学
- 医疗保健中的人工智能
- 临床预测建模临床预测建模
背景情况:
- 电子健康记录 (EHR) 包含大量用于临床预测的数据.
- 大型语言模型 (LLM) 是新兴的人工智能工具,在医疗保健领域具有潜力.
- 传统的机器学习 (ML) 是为医学中的预测任务而建立的.
研究的目的:
- 使用EHR数据,比较LLM (GPT-3.5,GPT-4) 与传统ML的临床预测疗效.
- 为了评估LLM的性能,校准,公平性和隐私的稳定性.
- 调查在语境学习对LLM绩效的影响.
主要方法:
- 在EHR数据 (VUMC,MIMIC IV) 上评估了GPT-3.5,GPT-4和梯度增强树 (ML).
- 使用AUROC测量预测性能,并使用Brier分数进行校准.
- 通过均等赔率和统计平价来评估公平性;通过将人口统计变量概括来评估隐私弹性.
主要成果:
- 传统的ML在预测性能和校准方面明显优于GPT-3.5和GPT-4.
- 机器学习模型在保护隐私的数据概括方面表现出更大的稳定性.
- GPT-4表现出更好的公平性,但性能受到了损害;LLM显示出比ML更差的校准.
结论:
- 当前的LLM在使用EHR数据进行临床预测任务时,效果和强度不如传统的ML.
- 虽然LLM很有前途,但需要进一步发展以匹配ML在这个领域的能力.
- 在LLMs的持续进步表明未来临床应用的潜力.
更多相关视频
04:09Predicting Treatment Response to Image-Guided Therapies Using Machine Learning: An Example for Trans-Arterial Treatment of Hepatocellular Carcinoma
Published on: October 10, 2018
8.2K
07:15Machine Learning Algorithms for Early Detection of Bone Metastases in an Experimental Rat Model
Published on: August 16, 2020
6.7K
相关概念视频
Language and Cognition
322
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
322
Steps in Outbreak Investigation
105
In the ever-evolving field of public health, statistical analysis serves as a cornerstone for understanding and managing disease outbreaks. By leveraging various statistical tools, health professionals can predict potential outbreaks, analyze ongoing situations, and devise effective responses to mitigate impact. For that to happen, there are a few possible stages of the analysis:
105
Clinical Trials
6.6K
Clinical trials are prospective experimental studies conducted on humans to determine the safety and efficacy of treatments, drugs, diet methods, and medical devices. Using statistics in clinical trials enables researchers to derive reasonable and accurate conclusions from the collected data, allowing them to make wise decisions in uncertain situations. In medical research, statistical methods are crucial for preventing errors and bias.
There are four phases in a clinical trial. A phase one...
There are four phases in a clinical trial. A phase one...
6.6K
Clinical Trials: Overview
2.7K
Clinical development focuses on how the drug will interact with the human body and encompasses four key phases of clinical trials, each serving a specific purpose in assessing the safety and effectiveness of new drugs. These phases overlap and build upon one another. Phase I involves a small group of healthy volunteers (typically 20-80 individuals) or, in cases where significant toxicity is expected, patients with the targeted disease, such as cancer or AIDS. The volunteers are tested for...
2.7K
