在产科系统性审查中评估六个大型语言模型用于研究识别
Shmuel Somer1, Maia Vardy2, Jonathan Eisenberger1
1Sheba Medical Center, Tel Hashomer, Israel.
Minerva obstetrics and gynecology
|January 29, 2026
概括
一般用途的大型语言模型 (LLM) 目前在系统性审查中无法用于自主研究识别. 人类监督对于产科和妇科的准确证据合成至关重要.
科学领域:
- 人工智能在医学中的应用
- 证据综合 证据综合
- 医疗信息学 医疗信息学
背景情况:
- 系统性审查需要根据复杂的标准准确确定研究.
- 大型语言模型 (LLM) 越来越多地被整合到证据合成中.
- 评估LLM可靠性对于研究识别至关重要.
研究的目的:
- 评估通用LLM在识别用于产科系统审查的研究中的准确性.
- 用预定义的资格标准来评估LLM绩效,而无需人类监督.
主要方法:
- 六个公开可访问的LLM被要求从已发表的元分析中确定研究.
- 将LLM的成果与参考审查中包含的14项研究进行了比较.
- 关键指标包括精度,回忆,F1得分和幻觉严重程度.
主要成果:
- 克劳德3.7显示了最高的准确性 (F1得分为0.48),识别了14项研究中的5项.
- 其他LLM的表现不佳 (F1得分为0.08-0.12),表明精度和回忆能力低.
- 来自LLM的更高的研究成果与幻觉增加相关.
结论:
- 目前的通用LLM在系统性审查中无法可靠地进行自主研究识别.
- 人类监督对于临床证据合成中准确的研究选择至关重要.
- 微调LLM和开发混合AI-人类工作流程是安全整合的必要条件.
相关概念视频
Language
909
Language is a unique communication system that uses words and systematic rules to organize and transmit information. Unlike other forms of communication, which may involve postures, movements, odors, or vocalizations, language relies on symbols and grammar. This makes human communication distinct from that of other species, who also communicate but do not use language in the same way humans do.
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
909
Review and Preview
8.4K
In statistics, several tools are used to interpret the data. Measures of central tendency represent the characteristics of the data, such as mean, median, and mode. Additionally, measures of variance like standard deviation and range are used to find the spread of data from the mean. Relative standing measures the distance between data locations. Commonly used measures of relative standings are percentile, z score, and quartiles.
Percentiles are a type of fractile that partition data into...
Percentiles are a type of fractile that partition data into...
8.4K
Review and Preview
11.2K
Data are individual items of information obtained from a population or sample. Data may be classified as qualitative (categorical), quantitative continuous, or quantitative discrete. Because it is not practical to measure the entire population in a study, researchers use samples to represent the population. A random sample is a representative group from the population chosen by using a method that gives each individual in the population an equal chance of being included in the sample. Random...
11.2K
Components of Language
817
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
817
Language Development
911
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
911
Language and Cognition
797
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
797


