AI-generated guidance for non-specialist physicians managing inflammatory bowel disease: expert evaluation and...

Shortages of inflammatory bowel disease (IBD) specialists, particularly in China and other resource‑constrained settings, mean that many patients are managed by non‑specialist physicians. Artificial intelligence (AI) models may provide clinical guidance, but their practical value for non‑specialists remains uncertain. We evaluated the quality of AI‑generated IBD guidance and its utility for non‑specialist physicians using standardized, case‑based clinical vignettes.

Twenty real-world IBD clinical questions submitted by non-specialist physicians and one standardized case vignette were entered into GPT-4 and DeepSeek using identical prompts. Seven IBD specialists independently rated response accuracy and completeness on 5-point Likert scales, and 12 non-specialist physicians rated clinical usefulness. The 12 non‑specialist physicians completed a two‑round cross‑over, pre-post case‑based test using 20 IBD clinical vignettes to evaluate whether reviewing AI outputs improves responses to common management vignettes.

Both models produced generally accurate and complete answers (median expert scores ≥4.0/5). Experts rated DeepSeek higher for the accuracy and completeness of diagnosis,treatment and, and follow-up. After reviewing AI answers, non‑specialist physicians' case‑based test scores improved significantly: median per‑round score (0-10) increased from 5.5 to 8.0 for GPT‑4 (Δ=3.0, p<0.001) and from 6.0 to 8.0 for DeepSeek (Δ=2.5, p<0.001). The between‑model difference in score change was not significant (p=0.781). Across both models, common limitations included limited patient‑level individualization, inconsistent red‑flag/referral triggers, and lack of verifiable, time‑stamped references; no harmful recommendations were identified, but safety omissions were present in both models.

GPT-4 and DeepSeek generated relatively accurate and comprehensive guidance and improved non-specialist physicians' performance in IBD case vignettes testing, supporting their potential as adjunct educational and reference tools for clinical reasoning. If implemented clinically, AI-based support should incorporate explicit safety prompts (red flags and referral triggers), and be validated prospectively within real-world workflows.