大型语言模型,科学知识和事实:简化人类专家评估的框架
Magdalena Wysocka1, Oskar Wysocki2, Maxime Delmas3
1Digital Cancer Research, CRUK National Biomarker Centre, Manchester, United Kingdom; Department of Computer Science, University of Manchester, Manchester, United Kingdom.
Journal of biomedical informatics
|September 14, 2024
概括
大型语言模型 (LLM) 显示出科学发现的希望,但在生物医学知识任务中与事实准确性作斗争. 域名专业化和增加的人类反可能会提高它们作为知识库的可靠性.
科学领域:
- 生物医学信息学 生物医学信息学
- 科学中的人工智能.
背景情况:
- 大型语言模型 (LLM) 通过处理大量的科学文献来加速生物医学发现的潜力.
- 目前的LLM应用程序在准确推断和提取复杂的生物医学信息方面面临着挑战.
研究的目的:
- 引入一个新的框架来简化对法学士编码的事实科学知识的评估.
- 评估11个最先进的LLM在生物医学知识任务,特别是抗生素发现方面的能力.
主要方法:
- 一个三步评估框架,评估流性,对齐性,连贯性,事实知识和特异性.
- 在非专家和领域专家之间分配任务,以优化评估效率.
- 对化学化合物定义和关系确定任务的LLM系统评估.
主要成果:
- 法律学士表现出更好的流利性,但事实准确性低,对过度代表的实体有偏见.
- 作为独立的生物医学知识库,LLM的可靠性是可疑的.
- 强调了对LLM产生的生物医学知识有系统评估框架的需要.
结论:
- 目前的LLM不适合零射击生物医学事实知识检索.
- 新兴的属性表明,随着领域专业化,模型规模的增加和人类反,事实性得到了改善.
更多相关视频
相关概念视频
Improving Translational Accuracy
9.4K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
9.4K
Accuracy, limits, and approximation
445
Accuracy, limits, and approximations are common in many fields, especially in engineering calculations. These concepts are imperative for ensuring that a given value is as close as possible to its true value.
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
445
Accuracy and Errors in Hypothesis Testing
183
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
183
Reliability and Validity
12.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
12.7K
Stereotype Content Model
14.0K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
14.0K
Heuristics
81
Heuristics are problem-solving strategies that use mental shortcuts to simplify decision-making. Unlike algorithms, which must be followed precisely to achieve a correct result, heuristics offer a general problem-solving framework. They save time and energy but can sometimes lead to less rational decisions.
People often rely on heuristics when faced with an overload of information, limited time, low importance of the decision, limited information, or when a heuristic readily comes to mind. For...
People often rely on heuristics when faced with an overload of information, limited time, low importance of the decision, limited information, or when a heuristic readily comes to mind. For...
81


