相关实验视频

Updated: Jun 4, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

在患者互动任务中使用大型语言模型的临床评估框架

Shreya Johri¹, Jaehwan Jeong^1,2, Benjamin A Tran³

¹Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.

Nature medicine

|January 3, 2025

概括

本研究介绍了CRAFT-MD,这是一种通过自然对话测试临床大语言模型 (LLM) 的新方法. 目前的LLM在诊断推理和准确性方面存在局限性,突出显示了医学更好的评估框架的需要.

科学领域:

人工智能在医学中的应用
临床决策支持系统临床决策支持系统
自然语言处理自然语言处理.

背景情况:

大型语言模型 (LLM) 显示出改变临床诊断和医生与患者互动的前景.
在现实世界中,LLM的准备和临床应用需要严格的标准化测试.
现有的评估方法往往缺乏自然医学对话的细微差别.

研究的目的:

引入用于医学测试的对话推理评估框架 (CRAFT-MD) 来评估临床LLMs.
在对话环境中评估著名的LLM (GPT-4,GPT-3.5,Mistral,LLaMA-2-7b) 的诊断能力和局限性.
为未来临床LLM评估提出建议.

主要方法:

开发和应用CRAFT-MD框架,利用模拟的人工智能代理进行基于自然对话的LLM互动.
对12个医学专业的LLM绩效进行评估.
评估基于文本和多式联络 (GPT-4V) 的对话和视觉能力.

主要成果:

确定了当前LLM在临床对话推理,病史记录和诊断准确性方面的重大局限性.
在各种LLM中观察到的局限性,甚至在多式联运能力的情况下也存在.
在不同的医学专业中,表现差距明显.

结论:

目前的LLM需要在广泛临床部署之前进行实质性改进.
CRAFT-MD框架提供了一种比传统方法更现实的方法来评估临床LLM.
建议强调现实的对话模拟,全面评估和混合评估方法来为未来的LLM测试.

更多相关视频

Foreign Accent and Forensic Speaker Identification in Voice Lineups: The Influence of Acoustic Features Based on Prosody

Foreign Accent and Forensic Speaker Identification in Voice Lineups: The Influence of Acoustic Features Based on Prosody

Published on: September 27, 2024

Objectification of Tongue Diagnosis in Traditional Medicine, Data Analysis, and Study Application

Objectification of Tongue Diagnosis in Traditional Medicine, Data Analysis, and Study Application

Published on: April 14, 2023

相关概念视频

Modeling in Therapy

Modeling in Therapy

Modeling, a key technique in therapy, uses observational learning to help clients acquire and practice new skills by watching therapists demonstrate desired behaviors. This approach, rooted in Albert Bandura's concept of vicarious learning, plays a significant role in therapeutic interventions for various psychological conditions, including social anxiety, ADHD, and depression.
Participant Modeling
Participant modeling involves therapists demonstrating calm and effective behaviors in...

Language and Cognition

Language and Cognition

Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.