自动报告器:开发一种人工智能工具,用于自动评估研究报告准则的遵守情况
David Chen1,2, Patrick Li3, Ealia Khoshkish2
1Princess Margaret Cancer Centre, Radiation Medicine Program, Toronto, ON M5G 2C4, Canada.
Journal of the American Medical Informatics Association : JAMIA
|December 23, 2025
概括
自动报告,一个大型语言模型系统,自动化研究报告准则遵守检查. 这种人工智能工具提供了高精度和与人类专家的实质性协议,改善了科学质量控制.
科学领域:
- 生物医学信息学 生物医学信息学
- 科学出版科学出版
- 人工智能的人工智能
背景情况:
- 遵守研究报告准则对于科学完整性和可重复性至关重要.
- 手动评估准则的遵守是耗时和资源密集的.
- 现有的自动化方法往往需要大量的训练数据和计算资源.
研究的目的:
- 开发AutoReporter,一个新的大型语言模型 (LLM) 系统,旨在自动评估对研究报告准则的遵守.
- 为了对基于LLM的准则遵守性评估进行各种提示工程和检索策略进行基准测试.
- 在多样化的数据集上验证开发系统的性能.
主要方法:
- 在SPIRIT-CONSORT-TM集体上使用推理和通用LLM评估了八种提示工程和检索策略.
- 性能最好的方法,AutoReporter,使用了零射击,无检索提示符与o3-mini推理LLM.
- 自动报告器在BenchReport上得到了验证,BenchReport是一个新的数据集,包括来自10次系统审查的专家评级评估.
主要成果:
- 自动报告器实现了高准确度:CONSORT的准确率为90.09%,SPIRIT的准确率为92.07%.
- 该系统与人类专家达成了实质性的一致性 (Cohen的 κ > 0.70对于CONSORT和SPIRIT).
- 在BenchReport数据集上的表现产生了91.8%的平均准确度和与专家评级的实质一致 (科恩的 κ > 0.6).
结论:
- 仅仅在AutoReporter中实现的结构化提示即可实现与微调的域特定模型相比或超过的性能.
- 这种方法消除了手动注释的 corpora 和计算密集训练的需要.
- 法律法规提供了一种可行的解决方案,用于自动化报告准则遵守评估,使科学研究中的可扩展质量控制成为可能.
相关概念视频
01:10Pre-Procedural Guidelines for Assessing Blood Pressure
788
Accurate blood pressure assessment is crucial for diagnosing and managing various health conditions. To ensure the reliability of these measurements, healthcare professionals must adhere to standardized pre-procedural guidelines. These guidelines enhance patient safety and improve the overall quality of healthcare. The following steps are essential for obtaining accurate and consistent blood pressure readings, from using the appropriate tools to ensuring effective communication with the...
788
01:16Non-equilibrium in the Cell
5.3K
An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...
5.3K
01:11Guidelines for Writing Outcome
3.6K
When developing expected outcomes for a patient care plan, the nurse should adhere to the following recommendations:
Patient outcomes reflect the patient's response to the goal rather than what the nurse aims to achieve. Terminology should be observable and measurable to avoid the reader's interpretation. The desired outcome should be realistic and achievable in the designated care timeframe. Expected outcomes should align with adjunctive therapies. The outcome should enhance care...
Patient outcomes reflect the patient's response to the goal rather than what the nurse aims to achieve. Terminology should be observable and measurable to avoid the reader's interpretation. The desired outcome should be realistic and achievable in the designated care timeframe. Expected outcomes should align with adjunctive therapies. The outcome should enhance care...
3.6K