Ordinal uncertainty grading in chest radiograph reports using large language models: a REFINE-aligned feasibility and methodology study

Radiologists differ in how strongly they hedge. We tested whether large language models can grade linguistic uncertainty in chest radiograph reports on a 0-3 scale and whether aggregate agreement conceals category-specific differences.

We used 3,927 public chest radiograph reports (Open-I, Indiana University). Three radiologists independently scored 100 reports from 0 (definitive wording) to 3 (extensive ambiguity). Majority vote, with prespecified attending adjudication for seven complete three-way disagreements, formed the reference standard. Claude Opus 4.5 and Gemini 2.5 Pro scored the same reports from a fixed prompt without task-specific training. We measured agreement using Cohen's quadratic weighted kappa (QWK). We also recorded per-score recall, ordinal error, prompt and reference-standard robustness, and comparison with a keyword-counting baseline. We then scored the remaining 3,827 reports with both models.

Human pairwise QWK ranged from 0.421 to 0.659. Claude and Gemini had similar aggregate agreement: QWK: 0.807 [95% confidence interval (CI), 0.722-0.876] and 0.814 (95% CI, 0.735-0.877), respectively; paired difference: 0.007 (95% CI, -0.062 to 0.079). In this validation sample, Claude identified 5/8 reports exactly as Score 3 and Gemini 1/8; Score-3 precision was 5/11 and 1/3, respectively. Of Gemini's seven exact Score-3 misses, six were assigned the adjacent Score 2 category. Reference-standard sensitivities yielded QWK: 0.810-0.841; prompt-variant pairwise agreement ranged from 0.818 to 0.913; repeat runs were identical for 29/30 Claude and 28/30 Gemini reports. All 3,827 Gemini discovery requests completed successfully, with token accounting recorded for the discovery cohort.

The approach is workable at the corpus scale, but similar aggregate agreement concealed differences in exact Score-3 classification, largely involving the Score-2/3 boundary. The score reflects wording, not clinical risk, and must not be used for risk triage. Clinical or large-scale use requires external validation enriched for ambiguous cases and prior radiologist calibration.

These scores may help radiologists audit reporting language and build balanced research evaluation sets, but downstream benefit must be demonstrated rather than assumed.