Reliable Detection, Unreliable Clinical Decision-making: Artificial Intelligence Models in Burn Assessment

Multimodal artificial intelligence (AI) systems are increasingly used for visual interpretation tasks. Whether commercially available models can be applied to burn surgery, and to what extent, remains unclear. We evaluated a clinical chain from visual recognition (distinguishing burns from nonburned skin) to treatment decision-making (debridement versus immediate skin grafting), including accuracy, output consistency, and skin tone subgroup performance.

Four multimodal large language models (GPT-5.4 Pro, Grok 4.1, Gemini 3.1 Pro, and Claude Opus 4.6) evaluated 50 cropped clinical photographs (25 burns and 25 nonburns) across 5 independent runs (n = 4000 total assessments). Models classified burn presence and treatment recommendation. Ground truth was established by expert consensus among plastic surgeons. Reproducibility was assessed using Fleiss κ.

Burn detection was near-perfect (97.2%-100%, κ = 0.896-1.000) and equivalent across Fitzpatrick skin tone strata. Treatment accuracy in burn cases ranged from 24.8% to 72.8%, with 3 models performing below the majority-class baseline. Reproducibility for treatment decisions dropped to κ = 0.132-0.385 in 3 models, with inconsistent recommendations for identical images. Error patterns were systematic and opposing across models.

AI systems reliably detected burns but failed to reliably recommend treatment. Reproducibility remained high for visual recognition but substantially lower for clinical decision-making. The clinical chain breaks at the transition from perception to decision, limiting the current applicability in burn surgery. The high sensitivity (94.4%-100%) and specificity (100%) for burn detection suggest that visual recognition may serve as a reliable foundation for future AI-assisted triage in burn care.