检索增强生成增强了日本整形外科委员会考试中的大型语言模型性能
Juntaro Maruyama1, Satoshi Maki2, Takeo Furuya1
1Department of Orthopedic Surgery, Graduate School of Medicine, Chiba University, Japan.
概括
检索增强代 (RAG) 显著改善了大型语言模型 (LLM),如医疗考试的GPT-4o和o1-预览. 然而,由于RAG的推理能力有限,RAG没有受益于GPT-3.5 Turbo.
科学领域:
- 人工智能在医学中的应用
- 自然语言处理自然语言处理.
- 医疗教育 技术 技术 医学教育
背景情况:
- 大型语言模型 (LLM) 在医疗保健方面表现有前途,但需要特定领域的准确性.
- 检索增强生成 (RAG) 旨在通过减少幻觉和改善获得专业信息来增强LLM.
- 在专业医疗领域RAG的有效性尚未完全理解.
研究的目的:
- 评估RAG对LLMs (GPT-3.5 Turbo,GPT-4o,o1-preview) 在专业医疗检查中的表现的影响.
- 确定RAG是否可以弥合医疗应用中LLM的准确性差距.
- 评估RAG增强的LLM对医疗用途的适用性.
主要方法:
- 从标准教科书中创建了一个专门的骨科数据库.
- 在日本骨科检查问题 (基于文本和图像) 上,GPT-3.5 Turbo,GPT-4o和o1-preview在RAG和无RAG的情况下进行了测试.
- 进行错误分析以确定性能决定因素.
主要成果:
- GPT-3.5 Turbo没有与RAG显著改善 (28%的精度与RAG相比29%没有).
- 使用RAG.o,GPT-4o准确度从62%增加到72%.
- o1-预览精度从67%大幅提高到84%的RAG.
结论:
- RAG显著提高了GPT-4o和o1-预览的性能,o1-预览获得了临床相关的熟练程度.
- GPT-3.5 Turbo的性能没有得到RAG的改善,这表明缺乏必要的推理能力.
- 在专业医疗领域,RAG集成显示了提高LLM准确性的潜力,特别是在先进模型中.
更多相关视频
03:14Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
相关概念视频
01:11Genetic Lingo
Overview
02:07Improving Translational Accuracy
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
02:07Improving Translational Accuracy
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
01:10Higher Mental Functions of the Brain: Language
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
01:22Language Development
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
01:12Retrieval
Retrieval is the process of getting information out of memory storage and back into conscious awareness. This ability is essential for daily tasks like brushing hair and teeth, driving to work, and performing job duties. Retrieval occurs in three ways: recall, recognition, and relearning.
Recall involves accessing information without cues, such as during an essay test, where individuals must retrieve facts and concepts from memory unaided. Another example is remembering the name of a colleague...
Recall involves accessing information without cues, such as during an essay test, where individuals must retrieve facts and concepts from memory unaided. Another example is remembering the name of a colleague...