没有准备好进入黄金时段:在观察性研究中评估偏差风险时,提取增长世代大语言模型的局限性
Samuel A Beber1, Katherine D Groff1, Tyler R Mange1
1Division of Pediatric Orthopaedic Surgery, Hospital for Special Surgery, New York, NY, USA.
检索增强型大语言模型 (RAG-LLMs) 在评估儿科骨科研究中显示出较低的准确性. 在使用人工智能工具进行研究时建议谨慎使用,直到进一步验证为止.
科学领域:
- 医疗研究中的人工智能
- 系统审查 增加 增加
- 儿科 整形医学 文献分析 文献分析
背景情况:
- 大型语言模型 (LLM) 越来越多地用于增强系统性审查.
- LLM可以"幻觉";检索增强生成 (RAG) 通过使用提供源来限制这种情况.
- 这项研究评估了RAG-LLM在对儿科骨科观察性研究的质量评估中的准确性.
研究的目的:
- 评估RAG-LLM (NotebookLM) 的准确性和可靠性,以评估儿科骨科观察性研究的质量.
- 为了比较RAG-LLM的性能与手动审查 (地面真相).
主要方法:
- 包括了两项关于儿科骨科观察研究的现有系统性审查.
- NotebookLM使用纽卡斯尔-太华尺度 (NOS) 评估了研究质量.
- 类内相关系数 (ICC) 测量RAG-LLM分数和手动审查分数之间的一致性.
主要成果:
- 总体而言,RAG-LLM实例与手动审查之间的一致性是中等的 (ICC{2,k) = 0.69).
- 个人RAG-LLM得分和手动审查之间的一致性很差 (ICC ((2,1) 从0.081到0.27不等).
- 百分比协议在14.8%至29.6%之间.
结论:
- 在质量评估方面,RAG-LLM (NotebookLM) 的可靠性和准确性较低.
- 研究人员在研究中使用RAG-LLM等AI工具时应谨慎使用.
- 研究人员的独立批判性评估对于新研究至关重要.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
07:31Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack
Published on: May 15, 2020
相关概念视频
Bias in Epidemiological Studies
Bias
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
Naturalistic Observations
Observational Studies
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One...
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Confounding in Epidemiological Studies
