相关实验视频

Updated: Sep 17, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

在使用大型语言模型的烟草研究中,用于检索增强生成的可扩展评估框架

Sherif Elmitwalli¹, John Mehegan², Sophie Braznell²

¹Tobacco Control Research Group, Department for Health, University of Bath, Bath, UK. se606@bath.ac.uk.

Scientific reports

|July 2, 2025

概括

一个新的框架评估了烟草研究的检索增强生成 (RAG) 中的大型语言模型 (LLM). 混合8×7B的表现优于Llama 3.1 70B,这表明需要在公共卫生领域进行特定领域的LLM评估.

科学领域:

人工智能的人工智能
公共卫生信息学公共卫生信息学
计算语言学计算语言学

背景情况:

专业知识领域需要对检索增强生成 (RAG) 系统进行强有力的评估.
烟草研究领域缺乏用于评估大型语言模型 (LLM) 的标准化框架,这阻碍了用于公共卫生决策的准确信息检索.
复杂的烟草行业文档需要特定领域的LLM评估.

研究的目的:

开发和验证RAG系统的烟草领域特定评估框架.
通过使用自动化指标和专家验证,在RAG配置中评估和比较不同LLM的性能.
为LLM在专业知识领域的表现建立一个基准.

主要方法:

在RAG系统中使用目标-问题-度量范式来评估LLMs (Mixtral 8×7B和Llama 3.1 70B).
包含使用GPT-4o的自动评估和三名烟草研究专家的验证.
采用了由20个精心策划的查询组成的域特定数据集,以衡量9个指标的性能,包括准确性,域特异性,完整性和清晰性.

主要成果:

开发的框架成功地区分了LLM架构之间的性能.
混合式8×7B在精度 (8.8/10对7.55/10) 和域特异性 (8.65/10对7.6/10) 中明显优于Llama 3.1 70B.
超参数优化提高了Mixtral的完整性,证明了框架对模型改进的实用性.

结论:

建立了一个强大的框架来评估烟草研究RAG系统中的LLMs,适用于其他专业领域.
强调在公共卫生应用中对LLM进行领域特定评估的关键重要性,原因是性能存在显著差异.
建议将框架扩展到更广泛的公司和额外的LLM进行全面评估.

关键词:

人工智能评估AI评估域特定信息检索的获取专家验证验证是专家的验证.目标-问题-度量框架大型语言模型获取 - 增强生成

更多相关视频

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

Objectification of Tongue Diagnosis in Traditional Medicine, Data Analysis, and Study Application

Objectification of Tongue Diagnosis in Traditional Medicine, Data Analysis, and Study Application

Published on: April 14, 2023

相关概念视频

Language Development

Language Development

Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...

Improving Translational Accuracy

Improving Translational Accuracy

Language and Cognition

Language and Cognition

Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.