Auto-METRICS: LLM-assisted scientific quality control for radiomics research.

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Radiomics

Background:

• Radiomics research quality is crucial for clinical translation but often suffers from methodological flaws.
• Standardized assessment tools like the METhodological RadiomICs Score (METRICS) are needed to ensure research integrity.

Purpose of the Study:

• To evaluate the reliability of large language models (LLMs) in assessing radiomics methodological quality using the METRICS framework.
• To compare LLM performance against human radiologists in a reproducibility setting.

Main Methods:

• A commercial LLM (Gemini Flash 2.0) and 24 open LLMs were used to assess the METRICS score for 46 radiomics articles.
• Assessments were compared with those of radiologists in two reproducibility studies (ADA2025 and K2025).
• Inter-rater reliability (Cohen's kappa) and scoring correlation (Pearson's correlation) were analyzed.

Main Results:

• The commercial LLM demonstrated inter-rater agreement with human radiologists comparable to human-human agreement in both studies (average kappa ≈ 0.48-0.58).
• METRICS scoring correlation between the LLM and human raters was also comparable to human-human correlation (average PC ≈ 0.62-0.68).
• An open, privacy-preserving LLM (Phi4-Reasoning) performed comparably to the commercial LLM.

Conclusions:

• LLMs can effectively assist in the standardized quality assessment of radiomics research.
• Open-source, privacy-preserving LLMs offer a viable alternative to commercial models for supporting radiomics research integrity evaluation.