大型语言模型可以成为共识工作组的可行工具吗? 腹腔直截镜专家共识小组的经验
Frank G Lee1, Ellen L Larson1, Jane Vermunt1
1Division of Colon and Rectal Surgery, Mayo Clinic, Rochester, Minnesota.
Diseases of the colon and rectum
|January 6, 2026
概括
与双子座和ChatGPT相比,OpenEvidence在一项关于腹腔直肠切除文献合成的研究中显示出优越的内容适当性和引用质量. 专家们无法区分聊天机器人生成的文本和人类共识.
科学领域:
- 医疗信息学 医疗信息学
- 人工智能在医学中的应用
- 手术文学综合的综合研究
背景情况:
- 最近一个关于ventral rectopexy的共识更新是由ventral rectopexy国际专家小组发表的.
- 这项研究评估了大型语言模型 (LLM) 在共识出版之前合成腹腔直性文献的能力.
研究的目的:
- 为了比较不同LLM的性能,包括ChatGPT-4o,Gemini 1.5 Pro和OpenEvidence,在生成与腹腔直皮检查相关的响应和引用方面.
- 使用专家小组的共识作为评估LLM绩效的基准.
主要方法:
- 评估了LLM的内容适当性,可读性,响应长度,引用编造和引用质量.
- 一个由结直肠外科医生组成的小组评估了LLM生成的文本是否适当,并试图将其与专家共识区分开来.
主要成果:
- OpenEvidence获得了最高的内容适当性 (3.5/5),超过了Gemini (3.0/5) 和ChatGPT (2.8/5).
- 聊天GPT表现出最高的语言和可读性,但伪造了53%的引用,而Gemini为12%,OpenEvidence为0%.
- 结肠直肠外科医生可以以55%的准确度识别聊天机器人生成的文本.
结论:
- 在内容适当性和引用质量方面,OpenEvidence优于Gemini和ChatGPT.
- 由LLM生成的文本无法从专家的共识中区分为专家.
- 在适当的披露和监督下,LLM可以作为未来共识组的有价值的研究工具.
相关概念视频
02:16Stereotype Content Model
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence categorization, a person will feel...
01:29Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...