临床大语言模型评估通过专家审查 (CLEVER):框架开发和验证
Veysel Kocaman1, Mustafa Aytuğ Kaya2, Andrei Marian Feier1
1John Snow Labs Inc, 16192 Coastal Highway, Lewes, DE, 19958, United States, +1 (302) 786-5227.
JMIR AI
|December 4, 2025
概括
一种新的评估方法CLEVER表明,一个专门的小型LLM在临床任务中表现优于GPT-4o. 这突显了医疗应用中医疗专用大语言模型 (LLM) 的潜力.
科学领域:
- 人工智能在医学中的应用
- 自然语言处理自然语言处理.
- 临床信息学 临床信息学
背景情况:
- 评估大型语言模型 (LLM) 是一个挑战,因为数据污染和基准任务和临床实践之间的差距.
- 现有的LLM评估方法,如公共基准和LLM-as-a-judge,受到数据问题和自我偏好的偏见的限制.
- 迫切需要强大的评估框架,反映现实世界的临床实用性.
研究的目的:
- 引入CLEVER (临床大语言模型评估-专家评审),这是评估医疗保健LLM的新方法.
- 使用执业医生对LLM进行盲目,随机,基于偏好的评估.
- 为了比较一般目的的LLM与医疗保健特定的LLM在临床任务上的表现.
主要方法:
- 采用CLEVER方法来比较GPT-4o与两个医疗保健特定的LLM (8B和70B参数).
- 对三个不同的临床任务进行了评估:文本总结,信息提取和回答问题.
- 执业医生提供基于偏好的评估,重点关注事实性,临床相关性和简洁性.
主要成果:
- 在关键临床维度中,医生在45%至92%的病例中更喜欢较小的,针对医疗保健的LLM而不是GPT-4o.
- 与GPT-4o相比,特定于医疗保健的LLM在事实性,临床相关性和简洁性方面表现优异.
- 在开放式的医疗问题回答方面,表现相似,这表明专业的LLM在上下文依赖的任务中表现出色.
结论:
- 特定于医疗保健的LLM可以在需要了解临床背景的任务中超过更大,通用目的的LLM.
- CLEVER方法为评估临床LLM提供了有效和可靠的方法,通过注释者间协议和相关性分析得到证实.
- 这项研究强调了专门的LLM和专家评审对于推进AI在医学中的重要性.
相关概念视频
Data Validation
6.3K
Data validation is an essential part of a comprehensive assessment. Validation is confirming or verifying and opening the door to gathering more assessment data as it clarifies vague or unclear data. The process of checking and verifying the collected information is called data validation. The primary purpose of data validation is to ensure data is as free from error, bias, and misinterpretation as possible.
Nursing assessment guides are generally based on holistic models rather than medical...
Nursing assessment guides are generally based on holistic models rather than medical...
6.3K
Language Development
806
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
806
Language and Cognition
693
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
693
Reliability and Validity
13.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.7K
Components of Language
723
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
723


