Comparing guideline adherence and readability: Artificial intelligence with deep learning versus specialized...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Cardiovascular Diseases

Background:

• Peripheral artery disease (PAD) presents a significant global health burden.
• Artificial intelligence (AI), including large language models (LLMs) and chain-of-thought (CoT) reasoning, offers innovative strategies for clinical decision support.
• This study evaluates the performance of AI versus human physicians in managing PAD cases.

Purpose of the Study:

• To compare the guideline adherence and readability of AI-generated responses against physician recommendations for a standardized PAD case.
• To assess the potential of LLMs as adjunct tools in clinical practice for PAD management.

Main Methods:

• A cross-sectional study involving 30 physicians and 13 LLM systems (10 standard, 3 CoT) from Latin America.
• Responses concerning diagnosis, treatment, risks, and prognosis were evaluated against the 2024 ACC/AHA PAD guideline.
• Readability was assessed using five Spanish indices, and guideline adherence was scored by three experts.

Main Results:

• Guideline adherence was comparable between physicians and LLMs (p=0.169), with CoT-LLMs showing the highest scores (9.7).
• LLMs more frequently recommended supervised exercise (84.6% vs 30.0%, p=0.002) and revascularization for quality of life (69.2% vs 20.0%, p=0.004).
• LLM responses exhibited lower readability (46.9 vs 51.4, p=0.012), while CoT-LLMs demonstrated superior inter-rater reliability (ICC=0.98).

Conclusions:

• LLMs exhibit comparable guideline adherence to physicians in PAD management, with CoT models performing exceptionally well.
• Divergent treatment preferences between AI and physicians highlight AI's potential as supplementary clinical tools.
• Further research is warranted to explore the integration of AI in clinical workflows for PAD.