对生物医学自然语言处理应用程序和建议进行大型语言模型的基准测试
Qingyu Chen1,2, Yan Hu3, Xueqing Peng1
1Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, New Haven, CT, USA.
Nature communications
|April 5, 2025
概括
大型语言模型 (LLM) 显示了生物医学自然语言处理 (BioNLP) 的潜力,但传统的微调通常会优于它们. 开源LLM需要进一步微调,以匹配性能,特别是复杂的推理任务.
科学领域:
- 生物医学信息学 生物医学信息学
- 自然语言处理自然语言处理.
- 人工智能的人工智能
背景情况:
- 生物医学文献的指数增长需要自动化知识提取.
- 生物医学自然语言处理 (BioNLP) 为手动策划挑战提供了解决方案.
- 大型语言模型 (LLM) 在专门的BioNLP任务中的有效性需要系统的评估.
研究的目的:
- 系统地评估领先的LLM (GPT,LLaMA) 在各种生物NLP任务上的表现.
- 为了比较LLM性能 (零射击,少数射击,微调) 与已建立的模型 (BERT,BART).
- 确定实际挑战,并为生物NLP的LLM应用提供见解.
主要方法:
- 在12个BioNLP基准和6种应用类型中评估了4个LLM.
- 对LLMs零射击,少数射击和微调策略的比较分析.
- 对LLM输出进行不一致性,缺失信息和幻觉的评估,以及成本分析.
主要成果:
- 像BERT/BART这样的传统微调模型通常在大多数BioNLP任务上超过了零或少射击的LLM性能.
- 闭源LLM,如GPT-4,在推理密集型应用程序 (如医疗问题答案) 中表现出卓越的能力.
- 开源LLM需要微调以弥合业绩差距,所有评估的LLM都表现出事实准确性和完整性的问题.
结论:
- 法律法规为BioNLP提供了一个有前途的途径,但其实际应用需要仔细考虑模型选择和微调策略.
- 微调对于优化BioNLP的LLM性能至关重要,特别是对于开源变体.
- 解决幻觉和缺失信息等挑战对于生物医学知识合成中可靠的LLM部署至关重要.
更多相关视频
07:50A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
Published on: September 20, 2018
15.7K
09:20Cloud-Based Phrase Mining and Analysis of User-Defined Phrase-Category Association in Biomedical Publications
Published on: February 23, 2019
8.6K
相关概念视频
02:07Improving Translational Accuracy
8.5K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
8.5K
02:28Leaky Scanning
5.0K
During most eukaryotic translation processes, the small 40S ribosome subunit scans an mRNA from its 5' end until it encounters the first start AUG codon. The large 60S ribosomal subunit then joins the smaller one to initiate protein synthesis. The location of the translation initiation is largely determined by the nucleotides near the start codon as there may be multiple translation initiation sites present on the mRNA. Marilyn Kozak discovered that the sequence RCCAUGG (where R...
5.0K