相关实验视频

Updated: Jun 15, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

通过各种任务和评估人员评估LLM的熟练程度

Tong Min Kim¹, Youngrong Lee¹, Chansik Kim^1,2

¹Department of Medical Informatics, College of Medicine, The Catholic University of Korea, Republic of Korea.

Studies in health technology and informatics

|August 23, 2024

概括

这项研究使用8个任务和17个不同的评估者评估了大型语言模型 (LLM),这是从以前的有限评估中显著扩展的. 调查结果突出了任务多样性和评估员多样性的对LLM绩效评估的影响.

科学领域:

人工智能的人工智能
自然语言处理自然语言处理.
计算语言学计算语言学

背景情况:

之前对大型语言模型 (LLM) 的评估受到有限数量的任务和评估者的限制.
这些先前的研究往往只关注一个领域,可能会扭曲结果,忽视更广泛的LLM能力.

研究的目的:

综合评估四种不同的大型语言模型 (LLM) 的熟练程度.
调查各种任务和广泛的评估者对LLM绩效指标的影响.

主要方法:

采用了八个不同的任务来探讨各种职能中的LLM能力.
利用来自多个领域的17名评估员来评估32个独特的LLM输出.
将评估方法与以前的研究进行比较,这些研究的任务和评估人员较少.

主要成果:

证明任务的数量和类型对LLM绩效评估有重大影响.
展示了不同领域的评估人员的多样性对于强有力的评估至关重要.
根据具体任务和评估员的专业知识,确定了LLM熟练程度的变化.

结论:

法学士评估需要一个多方面的方法,包括多样化的任务和跨学科的评估员.
这些发现强调了狭义评估的局限性,并倡导更广泛的评估策略.
未来的研究应该考虑这些因素,以确保准确和可概括的LLM绩效表征.

关键词:

人工智能的人工智能是人工智能.临床注意事项临床注意事项大型语言模型.

更多相关视频

Measuring Statistical Learning Across Modalities and Domains in School-Aged Children Via an Online Platform and Neuroimaging Techniques

Measuring Statistical Learning Across Modalities and Domains in School-Aged Children Via an Online Platform and Neuroimaging Techniques

Published on: June 30, 2020

Foreign Accent and Forensic Speaker Identification in Voice Lineups: The Influence of Acoustic Features Based on Prosody

Foreign Accent and Forensic Speaker Identification in Voice Lineups: The Influence of Acoustic Features Based on Prosody

Published on: September 27, 2024

相关概念视频

Improving Translational Accuracy

Improving Translational Accuracy

Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...

Language and Cognition

Language and Cognition

Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.