在AI中的协作智能:评估AI理事会在USMLE上的表现
Yahya Shaikh1, Zainab Asiyah Jeelani-Shaikh2, Muzamillah Mushtaq Jeelani3
1Department of Health Policy & Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America.
PLOS digital health
|October 9, 2025
概括
一个新的"人工智能理事会"框架使用多个大型语言模型 (LLM) 协作回答医疗许可证考试问题,显著提高了对单个模型的准确性和一致性.
科学领域:
- 人工智能的人工智能
- 自然语言处理自然语言处理.
- 医疗教育 技术 技术 医学教育
背景情况:
- 大型语言模型 (LLM) 中的随机下一个令牌生成导致响应变化,挑战知识评估的准确性.
- 现有的LLM框架往往在复杂,高风险的考试中难以保持一致和可靠的表现.
研究的目的:
- 引入和评估一个新的多代理框架",AI理事会",以提高知识评估中的LLM绩效.
- 评估协作AI决策在提高美国医学执照考试 (USMLE) 问题的准确性方面的有效性.
主要方法:
- 开发了一个名为"AI理事会"的框架,包括多个GPT-4实例,由"促进者AI"促进协作决策.
- 理事会框架应用于USMLE第1步,第2步CK和第3步的325个问题.
- 基于共识的答案与单个实例的GPT-4性能和地面真相进行了比较.
主要成果:
- 理事会取得了高的共识准确率:第一步97%,第二步93%的CK,第三步94%的CK,超过单个GPT-4模型的性能.
- 在最初的分歧的情况下,理事会审议导致了83%的时间正确的共识.
- 理事会纠正了53%的多数错误答案,错误到正确答案的变化是正确到不正确的变化的5倍.
结论:
- "人工智能理事会"框架有效地减少了响应变化,并提高了基于LLM的知识评估的准确性.
- 协同人工智能决策将响应变化从局限性转变为适应性推理和答案改进的优势.
- 这项研究展示了人工智能实施的新范式,强调了集体人工智能对孤立操作的力量.
相关概念视频
01:29Measures of Intelligence
8.3K
Psychologists measure intelligence by using standardized tests that produce a score known as the intelligence quotient or IQ. To understand IQ tests, it's important to recognize the key principles behind their construction: validity, reliability, and standardization.
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this;...
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this;...
8.3K
01:27Intelligence
8.4K
The term "intelligence" is complex because it refers to both behavior and individuals, and its interpretation varies across cultures. European Americans tend to link intelligence with reasoning and cognitive skills, while in Kenya, it is tied to responsible participation in family and social life. In Uganda, intelligence is seen as the ability to know the right actions and carry them out effectively, while the Iatmul people of Papua New Guinea associate it with the capacity to remember...
8.4K
01:24Triarchic Theory of Intelligence
9.8K
Robert Sternberg's triarchic theory of intelligence posits that intelligence is composed of three distinct but interrelated components: analytical, creative, and practical intelligence.
9.8K
01:23Wechsler's Contribution to Measures of Intelligence
2.0K
David Wechsler, a psychologist who worked with World War I veterans, developed a significant IQ test in 1939 called the Wechsler-Bellevue Intelligence Scale. This test was innovative because it combined several subtests that measured both verbal and nonverbal skills, reflecting Wechsler's belief that intelligence is a global capacity involving purposeful action, rational thinking, and effective interaction with the environment. This test later evolved into the Wechsler Adult Intelligence...
2.0K
01:14Decision Making: Traditional Method
5.1K
The process of hypothesis testing based on the traditional method includes calculating the critical value, testing the value of the test statistic using the sample data, and interpreting these values.
First, a specific claim about the population parameter is decided based on the research question and is stated in a simple form. Further, an opposing statement to this claim is also stated. These statements can act as null and alternative hypotheses, out of which a null hypothesis would be a...
First, a specific claim about the population parameter is decided based on the research question and is stated in a simple form. Further, an opposing statement to this claim is also stated. These statements can act as null and alternative hypotheses, out of which a null hypothesis would be a...
5.1K
01:29Environmental Influences on Intelligence
886
Despite the strong genetic influence on traits like intelligence, environmental factors significantly shape outcomes. For example, while over 90% of height variation is due to genetic differences, environmental factors such as nutrition also have a notable impact. Similarly, for intelligence, changes in a child's surroundings can significantly alter their IQ. Research shows that enriched environments boost children's academic success and help them develop key cognitive skills. Children...
886