Comparing guideline adherence and readability: Artificial intelligence with deep learning versus specialized...

Peripheral artery disease (PAD) is a global health challenge. Advances in artificial intelligence (AI), such as large language models (LLMs) and chain-of-thought (CoT) reasoning, offer novel approaches for clinical recommendations. This study compared the readability and guideline adherence of responses from physicians and AI for a standardized PAD case.

This cross-sectional study gathered responses from 30 specialized physicians (11 cardiologists, 19 vascular surgeons) across seven Latin American countries and 13 LLM systems (10 standard, three CoT). Both groups addressed diagnosis, treatment, risks, and prognosis; LLMs responded as vascular specialists. Responses were blindly evaluated with five validated Spanish readability indices and compared to the 2024 ACC/AHA multisocietal PAD guideline. Three experts scored guideline adherence; nonparametric tests were applied.

Guideline adherence did not differ significantly between physicians (median 5.8 [3.4-7.6]) and LLMs (7.3 [4.7-9.7], p = 0.169), though CoT-LLMs achieved the highest scores (9.7 [8.5-11.0]). LLMs more often recommended supervised exercise (84.6% vs 30.0%, p = 0.002) and revascularization for quality of life (69.2% vs 20.0%, p = 0.004), whereas physicians favored cilostazol (60.0% vs 30.8%, p = 0.104). LLM responses had lower Readability μ values (46.9 vs 51.4, p = 0.012). Inter-rater reliability was highest for CoT-LLMs (intraclass correlation coefficient [ICC] = 0.98) versus physicians (ICC = 0.76).

LLM showed comparable guideline adherence to physicians although CoT models achieved the highest scores. The difference in physician and AI treatment preferences suggest the potential of AI as adjunct clinical tools and warrants further study.