在神经外科医生的自我评估中评估大型语言模型的性能和脆弱性
Krithik Vishwanath1,2,3, Anton Alyakin1,4, Mrigayu Ghosh5,6
1Department of Neurological Surgery, NYU Langone Health, New York, New York, USA.
Neurosurgery
|December 8, 2025
概括
大型语言模型 (LLM) 对神经外科考试有希望,但与分心作斗争. 测试显示,虽然一些LLM通过神经外科董事会类似的问题,但当不相关的信息存在时,他们的准确性显著下降.
科学领域:
- 人工智能在医学中的应用
- 神经外科教育 神经外科教育
- 自然语言处理自然语言处理.
背景情况:
- 神经外科住院医生使用神经外科医生大会神经外科医生自我评估委员会考试准备.
- 大型语言模型 (LLM) 越来越多地因其神经外科知识而被评估.
- 临床文本容易受到来自生成AI和环境指令的分心.
研究的目的:
- 评估最先进的LLM在神经外科董事会类问题上的表现.
- 为了评估LLM对文本中分散注意力的陈述的稳定性.
主要方法:
- 用2904个神经外科委员会考试问题评估了28个LLM.
- 引入了一个分散注意力的框架,其中包含多种词的无关陈述.
- 评估了由于干扰因素导致模型性能下降的情况.
主要成果:
- 在28个LLM中,有6个获得了通过分数.
- 干扰使得准确度降低了高达20.4%,导致一个模型失败.
- 开源模型的性能下降比专有模型的性能下降更大.
结论:
- 目前的LLM可以回答类似于神经外科董事会的问题,但容易受到分心的影响.
- 制定缓解策略对于安全的临床部署LLMs至关重要.
- 需要增强LLM对文本中干扰的弹性.
更多相关视频
13:12Translational Brain Mapping at the University of Rochester Medical Center: Preserving the Mind Through Personalized Brain Mapping
Published on: August 12, 2019
46.3K
03:14Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
994
相关概念视频
01:10Higher Mental Functions of the Brain: Language
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
01:29Self-Evaluation Maintenance Model
The Self-Evaluation Maintenance (SEM) model offers a psychological framework to understand how individuals’ self-esteem is influenced by the achievements of others, particularly those with whom they share close personal bonds. The SEM model operates when personal rather than social identity guides individuals. Central to this model is the notion that individuals have an inherent desire to preserve a favorable self-image, which is continuously shaped by interpersonal comparisons and...