Development and External Validation of a Large Language Model-Based Clinical Decision-Support System for Colonoscopy...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Gastroenterology

Background:

• Adherence to guideline-based colonoscopy surveillance intervals is suboptimal.
• Prior studies on automating interval assignment used proprietary models with limited generalizability.
• A novel clinical decision support system (CDS) was developed using open-source large language models (LLMs) and the 2020 U.S. Multi-Society Task Force (USMSTF) guidelines.

Purpose of the Study:

• To develop and externally validate a CDS for automating colonoscopy surveillance interval assignment.
• To assess the generalizability and performance of an open-source LLM-based approach.
• To improve adherence to guideline-based surveillance intervals.

Main Methods:

• Assembled development, internal, and external validation cohorts of colonoscopy-pathology data.
• Developed a three-stage CDS: LLM-based extraction of pathology report sections, LLM-based extraction of surveillance variables, and deterministic interval assignment using USMSTF logic.
• Benchmarked seven open-source LLMs, selecting the highest-performing model (Gemma-27B-Instruct) for integration.

Main Results:

• The CDS achieved 94.0% guideline-concordant interval assignment in internal validation and 92.8% in external validation.
• The best-performing open-source LLM, Gemma-27B-Instruct, demonstrated high accuracy (macro-F1: 0.992).
• Misclassifications were mainly due to adenoma miscounting or misinterpretation of biopsies; hallucination errors were <1%.

Conclusions:

• An externally validated, hybrid LLM-rules-based CDS for colonoscopy surveillance was successfully developed.
• The system achieves >90% concordance with established guidelines.
• This approach preserves transparency, auditability, and guideline traceability in automated surveillance assignment.