预训练语言模型的上下文差异评估,用于基于提示的生物医学知识探测
Zonghai Yao1, Yi Cao1, Zhichao Yang1
1College of Information and Computer Science, University of Massachusetts Amherst, Amherst, MA, USA.
概括
这项研究增强了预训练语言模型 (PLM) 的生物医学知识探测. 新的方法改善了复杂关系的评估,使BioLAMA更可靠地评估PLM理解.
科学领域:
- 计算语言学计算语言学
- 生物医学信息学是生物医学信息学.
- 人工智能的人工智能是人工智能.
背景情况:
- 预训练语言模型 (PLM) 越来越多地被研究为他们的学习知识.
- 填空任务,如cloze测试,对于知识评估是常见的.
- 像BioLAMA这样的现有基准因即时偏见和数据分布而面临局限性.
研究的目的:
- 解决BioLAMA中基于提示的知识探测的不可靠性和不稳定性.
- 开发改进的方法来评估生物医学领域的PLM事实知识.
- 引入一种新的评估指标,更好地捕捉到PLM的理解,而不仅仅是简单的回忆.
主要方法:
- 在快速生成中引入了上下文差异,以减轻探测偏差.
- 提出了一个新的基于等级变化的评估指标来评估PLM知识.
- 引入了"误解"的概念,以区分真正的理解与记忆.
主要成果:
- 对12个PLM的实验表明,上下文差异提示增强了BioLAMA适合大N-M和罕见关系的适用性.
- 拟议的理解-混-误解 (UCM) 度量表在评估PLM方面被证明是有效的.
- 控制实验有助于将真正的理解与简单的数据复制脱而出.
结论:
- 开发的上下文差异提示和UCM指标为PLM中的生物医学知识提供了更强大和可靠的评估.
- 这些进步解决了以前方法的关键局限性,特别是对于复杂和长尾的生物医学数据.
- 这些发现有助于更深入地了解PLM真正获得的知识以及如何有效地衡量它.
相关概念视频
Improving Translational Accuracy
11.7K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
11.7K
Language and Cognition
377
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
377
Language Development
403
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
403
Components of Language
329
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
329
Higher Mental Functions of the Brain: Language
934
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
934
Pharmacokinetic Models: Overview
813
Pharmacokinetic models utilize mathematical analysis to achieve a detailed quantitative understanding of a drug's life cycle within the body. They are instrumental in simulating a drug's pharmacokinetic parameters, predicting drug concentrations over time, optimizing dosage regimens, linking concentrations with pharmacologic activity, and estimating potential toxicity.
There are three primary types of models: empirical, compartment, and physiological. Empirical models, with minimal...
There are three primary types of models: empirical, compartment, and physiological. Empirical models, with minimal...
813


