系统性基准测试表明,大型语言模型尚未达到传统罕见病决策支持工具的诊断准确性
Justin T Reese1,2, Leonardo Chimirri2,3, Yasemin Bridges2,4
1Division of Environmental Genomics and Systems Biology, Lawrence Berkeley National Laboratory, Berkeley, CA, USA.
medRxiv : the preprint server for health sciences
|August 7, 2024
概括
大型语言模型 (LLM) 显示出诊断遗传疾病的潜力,但落后于传统工具. 基准测试显示,LLM在23.6%的病例中正确识别了诊断,而Exomiser的情况为35.5%.
科学领域:
- 计算生物学是一种计算生物学.
- 医疗信息学医学信息学
- 遗传学 是一个遗传学.
背景情况:
- 大型语言模型 (LLM) 提供了在差异诊断中的临床决策支持的潜力.
- 在医学诊断中评估LLM的表现是具有挑战性的,因为输出是非结构化的.
- 遗传疾病诊断依赖于整合复杂的表型和基因型数据.
研究的目的:
- 为基因疾病基因预训变压器 (GPT) 模型的诊断能力进行基准测试.
- 为了将LLM的表现与已建立的诊断工具进行比较,Exomiser.
- 评估LLM在临床诊断工作流程中整合的准备程度.
主要方法:
- 利用了5,213个基因疾病病例报告,这些病例报告以Phenopacket Schema进行了结构化.
- 采用人类现象型本体学 (HPO) 和世界疾病本体学进行标准化.
- 为三个GPT模型生成了来自phenopackets的提示,并以仅表型模式运行Exomiser进行比较.
主要成果:
- 排名最高的LLM在23.6%的案例中将正确的遗传疾病诊断排在第一位.
- 传统的生物信息学工具Exomiser在35.5%的病例中实现了第一级诊断.
- 虽然LLM的表现有所改善,但与Exomiser的表现并没有匹配.
结论:
- 目前的LLM还没有满足已建立的生物信息学工具对遗传疾病的诊断准确性.
- 需要进一步的研究,以优化LLM整合到诊断管道.
- 法律法规表现有前途,但需要发展以克服非结构化响应评估的局限性.
相关概念视频
01:15Receiver Operating Characteristic Plot
109
A ROC (Receiver Operating Characteristic) plot is a graphical tool used to assess the performance of a binary classification model by illustrating the trade-off between sensitivity (true positive rate) and specificity (false positive rate). By plotting sensitivity against 1 - specificity across various threshold settings, the ROC curve shows how well the model distinguishes between classes, with a curve closer to the top-left corner indicating a more accurate model. The area under the ROC curve...
109
01:13Sensitivity, Specificity, and Predicted Value
236
In healthcare diagnostics, laboratory tests play a crucial role in identifying and diagnosing a wide range of medical conditions. However, interpreting test results is not always straightforward. An abnormal test result does not always confirm the presence of a disease, just as a normal result does not guarantee its absence. To assess the reliability of these diagnostic tools, healthcare practitioners rely on two key statistical indicators: sensitivity and specificity.
Sensitivity is the...
Sensitivity is the...
236