Would Artificial Intelligence Make 'Good' Decisions About Thyroid Nodule Biopsies?

Large language models (LLMs) have shown promise for clinical decision support (CDS), but evaluation under clinical equipoise-scenarios lacking a consensus correct answer-remains rare. We evaluated commercially available LLM performance for thyroid nodule biopsy recommendations under clinical equipoise.

One hundred clinical vignettes describing patients with incidentally discovered thyroid nodules of varying sizes and risk features were presented to 94 clinicians, recruited by convenience sampling from professional networks, and multiple LLMs (GPT-5, Claude Sonnet 3.5/4.0/4.5, Llama 3.2-90B, Deepseek R1, and OpenEvidence). Responses were dichotomized as biopsy versus no biopsy. Agreement was assessed using Krippendorff's alpha, and a deterministic algorithm served as a provisional benchmark. Selected LLM responses were reviewed by experts.

Clinicians recommended biopsy in 58% of responses (95% CI 50%, 66%), with unanimous responses in only 50 vignettes. LLM biopsy rates ranged from 46% (Claude 3.5) to 74% (Llama 3.2-90B). LLM-clinician agreement was moderate (α = 0.372-0.453). Agreement with the deterministic algorithm ranged from high (α = 0.954, Detailed TIRADS GPT-5) to low (α = 0.265, Claude Sonnet 4.5). Llama 3.2-90B recommended biopsy in 8 of 8 cases where clinicians unanimously recommended against it.

LLMs differed meaningfully in biopsy recommendation patterns, with potential population-level consequences. Evaluating and aligning LLMs for CDS under equipoise requires stakeholders and processes analogous to clinical guideline committees. Because LLMs used as CDS could powerfully disseminate different practice patterns, this creates new responsibilities for guidelines committees and other stakeholders.