An Artificial Intelligence Application for Standardized Expert-Level Interpretation of Vascular Clinical Practice...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Medical Informatics

Background:

• Clinical Practice Guidelines (CPGs) often present interpretation challenges for readers.
• Developing standardized methods for CPG interpretation is crucial for consistent clinical practice.
• The VascLink-AI application was created to address these interpretation complexities.

Purpose of the Study:

• To develop and validate an AI-driven application for standardized interpretation of Clinical Practice Guidelines (CPGs).
• To assess the performance of the VascLink-AI application against human expert interpretation.
• To evaluate the accuracy, completeness, clarity, relevance, adaptability, and evidence justification of AI-generated versus human-generated guideline interpretations.

Main Methods:

• A comparative study benchmarked AI performance against clinical standards using the 2017 ESC/ESVS Peripheral Arterial Diseases guidelines.
• Clinical guidelines were vectorized into a knowledge graph, and a large language model was trained on this graph.
• Performance was evaluated on 41 vignettes, with answers scored by an automated evaluator (GPT-4o) and two human experts, followed by non-inferiority analysis.

Main Results:

• VascLink-AI demonstrated superior performance compared to expert committee answers, with composite scores of 90.5% (automated) and 94.2% (human consensus) versus 77.8% and 86.2%, respectively (p<0.001).
• AI answers showed moderate inter-rater reliability (Kappa = 0.450), contrasting with negligible human rater agreement on committee answers (Kappa = -0.018).
• The application was found to be non-inferior to human experts across all 12 evaluated metrics (p < .001).

Conclusions:

• AI can evolve beyond a general research tool into a structured, guideline-specific framework.
• VascLink-AI successfully provides interpretations of societal guidelines that align with human expert understanding.
• This validates the potential of AI for enhancing consistency and accuracy in clinical guideline interpretation.