Development and External Validation of a Large Language Model-Based Clinical Decision-Support System for Colonoscopy...

Adherence to guideline-based colonoscopy surveillance intervals remains suboptimal. Large language models show promise for automating interval assignment, but prior studies relied on proprietary models and have variably assessed generalizability. We developed and externally validated a clinical decision-support system that integrates locally deployable, open-source large language model-based variable extraction with a rules engine that encodes the 2020 United States Multi-Society Task Force guidelines.

We assembled development (n = 256), internal (n = 450), and external validation (n = 415) cohorts of colonoscopy-pathology dyads. We developed a 3-stage clinical decision-support system including: (1) large language model-based extraction of diagnosis-bearing sections from pathology reports; (2) large language model-based extraction of surveillance-relevant variables; and (3) deterministic assignment of surveillance intervals using United States Multi-Society Task Force logic. Seven open-source large language models (3.8-32B parameters) were benchmarked; the highest-performing model by macro-averaged F1 score was locked prior to validation. The primary outcome was guideline-concordant interval assignment (target, 90%); secondary outcomes included extraction performance and hallucination rate.

The best-performing open-source model, Gemma-27B-Instruct (macro-averaged F1 score, 0.992), was locked. The clinical decision-support system achieved 94.0% guideline-concordant surveillance interval assignment in internal validation (95% confidence interval, 91.8%-96.0%; κ = 0.92; 95% confidence interval, 0.89-0.95) and 92.8% concordance in external validation (95% confidence interval, 90.1%-95.2%; κ = 0.88; 95% confidence interval, 0.84-0.91). Misclassifications were primarily due to adenoma miscounting and diagnostic biopsies misinterpreted as resections. Hallucination-related errors occurred in fewer than 1% of cases.

We report an externally validated, hybrid large language model rules-based clinical decision-support system for colonoscopy surveillance that achieves >90% concordance while preserving transparency, auditability, and guideline traceability.