AutoReporter: 研究報告ガイドライン遵守の自動評価のための人工知能ツールの開発

David Chen^1,2, Patrick Li³, Ealia Khoshkish²

¹Princess Margaret Cancer Centre, Radiation Medicine Program, Toronto, ON M5G 2C4, Canada.

Journal of the American Medical Informatics Association : JAMIA

|December 23, 2025

要旨

Objectives:

To develop AutoReporter, a large language model (LLM) system that automates evaluation of adherence to research reporting guidelines.

Materials And Methods:

Eight prompt-engineering and retrieval strategies coupled with reasoning and general-purpose LLMs were benchmarked on the SPIRIT-CONSORT-TM corpus. The top-performing approach, AutoReporter, was validated on BenchReport, a novel benchmark dataset of expert-rated reporting guideline assessments from 10 systematic reviews.

Results:

AutoReporter, a zero-shot, no-retrieval prompt coupled with the o3-mini reasoning LLM, demonstrated strong accuracy (CONSORT 90.09%; SPIRIT: 92.07%), substantial agreement with humans (CONSORT Cohen's κ = 0.70, SPIRIT Cohen's κ = 0.77), runtime (CONSORT: 617.26 s; SPIRIT: 544.51 s), and cost (CONSORT: 0.68 USD; SPIRIT: 0.65 USD). AutoReporter achieved a mean accuracy of 91.8% and substantial agreement (Cohen's κ > 0.6) with expert ratings from the BenchReport benchmark.

Discussion:

Structured prompting alone can match or exceed fine-tuned domain models while forgoing manually annotated corpora and computationally intensive training.

Conclusion:

Large language models can feasibly automate reporting guideline adherence assessments for scalable quality control in scientific research reporting. AutoReporter is publicly accessible at https://autoreporter.streamlit.app.

キーワード:

遵守一致大規模言語モデル品質管理報告ガイドライン

さらに関連する動画

08:58

Artificial Intelligence Approaches to Assessing Primary Cilia

Published on: May 1, 2021

4.1K

関連する概念動画

01:10

Pre-Procedural Guidelines for Assessing Blood Pressure

788

Accurate blood pressure assessment is crucial for diagnosing and managing various health conditions. To ensure the reliability of these measurements, healthcare professionals must adhere to standardized pre-procedural guidelines. These guidelines enhance patient safety and improve the overall quality of healthcare. The following steps are essential for obtaining accurate and consistent blood pressure readings, from using the appropriate tools to ensuring effective communication with the...

788

01:16

Non-equilibrium in the Cell

5.3K

An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...

5.3K

01:11

Guidelines for Writing Outcome

3.6K

When developing expected outcomes for a patient care plan, the nurse should adhere to the following recommendations:
Patient outcomes reflect the patient's response to the goal rather than what the nurse aims to achieve. Terminology should be observable and measurable to avoid the reader's interpretation. The desired outcome should be realistic and achievable in the designated care timeframe. Expected outcomes should align with adjunctive therapies. The outcome should enhance care...

3.6K