A multi-guideline clinical agent for vascular surgery: architecture, implementation, and benchmark evaluation
Vangelis G Alexiou1,2, Areti Vassiliou3, Bauer E Sumpio4
1Department of Vascular Surgery, University of Patras School of Medicine, Patras, Greece.
Background:
The clinical practice guidelines (CPGs) published by the European Society for Vascular Surgery (ESVS) span approximately 1100 pages whose simultaneous cross-referencing is impractical at the point of care. Unguarded large language models (LLMs) carry hallucination risk, motivating a guideline-locked design.
Methods:
We describe the architecture and development of a context-aware clinical agent encompassing the full ESVS corpus of 14 CPGs. The system extends a validated single-guideline GraphRAG prototype with an LLM-driven Guideline Router, an embedded Context Gate that withholds retrieval when decision-critical parameters are absent, dual-channel retrieval, and GPT-5 synthesis under a mandatory citation template. An automated benchmark assessed the final system on 55 vignettes, scored by 2 blinded LLM judges over 3 runs (330 judgments), with clinician adjudication of every flagged claim.
Results:
The system stayed within its locked corpus, declined all out-of-scope queries, and achieved 98.4% guideline-routing recall. The Context Gate showed 91.9% specificity and 78.6% sensitivity for underspecified queries, and all 259 cited recommendations were verifiable within the locked corpus. Automated judges flagged an unsupported claim in 18.2% of items under a sensitive screening definition; on adjudication by 2 vascular surgeons, 7.3% of items contained a clinically consequential error, none graded capable of causing harm. Clinical acceptability of complete answers was not scored and is reserved for prospective expert evaluation.
Conclusion:
These results support the feasibility of a multi-guideline context-aware clinical agent for vascular surgery; citation traceability and locked-corpus provenance facilitate future auditability.
