用于自动化临床试验标准的大型语言模型 转换为观察性医学结果 合作伙伴关系 常见数据模型查询:验证和评估 研究研究
Kye Hwa Lee1, Sujung Jang2, Grace Juyun Kim3
1Department of Information Medicine, Department of Digital Medicine, Asan Medical Center, University of Ulsan College of Medicine, 88 Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Republic of Korea, 82 10-3010-5991, 82 2-3010-2531.
自动化临床试验资格标准转换到SQL查询是具有挑战性的. 像Lama3:8b这样的较小模型表现出更高的潜力,在较低幻觉率的较大模型中表现出更高的潜力,尽管验证仍然至关重要.
科学领域:
- 医疗信息学 医疗信息学
- 人工智能在医学中的应用
- 临床试验设计 临床试验设计
背景情况:
- 现实世界的数据增强了临床试验设计.
- 自动化适用性标准转换为数据库查询面临准确性和可用性挑战.
研究的目的:
- 开发一个自动化系统,将ClinicalTrials.gov的自由文本符合性标准转换为OMOP CDM兼容的SQL查询.
- 在多个大型语言模型 (LLM) 中评估幻觉模式,以确定最佳的部署策略.
主要方法:
- 实施了三阶段的预处理管道 (细分,过,简化) 以保护临床语义.
- 用357个临床术语对比了GPT-4和USAGI的概念映射准确性.
- 使用SynPUF数据分析了8个LLM和5个提示策略的760次SQL生成尝试.
- 使用OMOP CDM数据库对已建立概念集进行验证生成的SQL查询.
主要成果:
- GPT-4实现了48.5%的概念绘图准确度,超过了USAGI (32.0%).
- 开源的 llama3:8b 模型显示了最高的有效 SQL 率 (75.8%),超过了 GPT-4 (45.3%),因为幻觉率较低 (21.1%).
- 总体而言,幻觉率为32.7%,常见的错误包括错误的域赋值和位符插入.
- 临床验证显示了可变的性能,对1型糖尿病的一致性很高 (Jaccard=0.81),但对2型糖尿病的一致性很小 (Jaccard=0.03).
结论:
- 法律法规可以加快资格标准的转换,但幻觉率需要仔细的模型选择和验证.
- 像lama3:8b这样的更小,更具成本效益的模型可以超过更大的商业LLMs.
- 未来的研究应该探索混合方法,将LLMs与复杂的临床概念的基于规则的方法相结合.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
07:50A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
Published on: September 20, 2018
相关概念视频
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Clinical Trials
There are four phases in a clinical trial. A phase one...
Methods of Documentation VI: Case Management Model
For example, a patient with a chronic...
Statistical Software for Data Analysis and Clinical Trials
Data Validation
Nursing assessment guides are generally based on holistic models rather than medical...
Observational Studies
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One...
