评估生物医学推理和分类模型的ChatGPT家族
Shan Chen1,2, Yingya Li3, Sheng Lu4
1Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA 02115, United States.
Journal of the American Medical Informatics Association : JAMIA
|January 23, 2024
概括
微调的BioBERT在生物医学分类和推理任务中表现优于ChatGPT等大型语言模型 (LLM). 尽管LLM的进步,专门的模型微调仍然是这些生物医学应用的优越策略.
科学领域:
- 生物医学自然语言处理
- 医疗保健中的人工智能
背景情况:
- 大型语言模型 (LLM) 在生物医学问答方面表现有前途,但对于专业应用需要进一步研究.
- 除了问答之外,评估LLM对于促进其在生物医学领域的实用性至关重要.
研究的目的:
- 调查ChatGPT家族模型 (GPT-3.5,GPT-4) 在生物医学分类和推理任务上的性能.
- 为了比较LLM的性能与传统方法,如字袋 (BoW) 和微调的BioBERT模型.
主要方法:
- 评估了11122个生物医学样本的模型性能,用于分类 (n=8676) 和推理 (n=2446).
- 利用了零和少数镜头设置的快速开发,包括思维链 (CoT).
- 通过逻辑回归和微调的BioBERT模型,与BoW比较最好的LLM提示.
主要成果:
- 微调的BioBERT在分类 (F1:0.800-0.902) 和推理 (F1:0.851) 中取得了最高的性能.
- 短暂的CoT LLM方法产生了与BoW模型相比的分类 (F1: 0.671-0.770) 和推理 (F1: 0.682) 结果.
- 与BioBERT (0.078小时) 和BoW (0.008小时) 相比,LLM提示工程需要相当长的时间 (78小时).
结论:
- 微调像BioBERT这样的专业模型仍然是基本的生物医学NLP任务最有效的策略.
- 尽管在LLM方面取得了进展,但针对复杂任务的快速工程需要大量的资源,并且与微调相比,产生了不够优的结果.
相关概念视频
Pedigree Analysis
Overview
Pedigree Analysis
Overview
Evolutionary Relationships through Genome Comparisons
Genome comparison is one of the excellent ways to interpret the evolutionary relationships between organisms. The basic principle of genome comparison is that if two species share a common feature, it is likely encoded by the DNA sequence conserved between both species. The advent of genome sequencing technologies in the late 20th century enabled scientists to understand the concept of conservation of domains between species and helped them to deduce evolutionary relationships across diverse...
Phylogenetic Trees
Phylogenetic trees come in many forms. It matters in which sequence the organisms are arranged from the bottom to the top of the tree, but the branches can rotate at their nodes without altering the information. The lines connecting individual nodes can be straight, angled, or even curved.The length of the branches can depict time or the relative amount of change among organisms. For instance, the branch length might indicate the number of amino acid changes in the sequence that underlies the...
Genetic Screens
Genetic screens are tools used to identify genes and mutations responsible for phenotypes of interest. Genetic screens help identify individuals or a group of people at risk of developing genetic diseases and help them with early intervention, targeted therapy, and reproductive options.
Forward genetic screens
Forward or “classical” genetic screens involve creating random mutations in an organism’s DNA using radiation, mutagens, or insertion of additional bases, which result in visible changes...
Forward genetic screens
Forward or “classical” genetic screens involve creating random mutations in an organism’s DNA using radiation, mutagens, or insertion of additional bases, which result in visible changes...
Microbial Classification System
Classification is the process of organizing organisms into hierarchically inclusive groups based on their phenotypic similarities or evolutionary relationships. A species comprises one or more strains, and closely related species are grouped into genera. Genera are further classified into families, families into orders, orders into classes, and so forth, up to the domain level, which is the broadest taxonomic rank derived from a combination of phenotypic and genotypic data.The nomenclature of...


