UMEval: a unified framework for explainable medical term semantic evaluation with large language models

Accurate medical term semantic evaluation is critical for patient safety, timely diagnosis, and healthcare interoperability. Existing methods suffer from incomplete knowledge coverage, insufficient interpretability, and unreliable output. To address these limitations, we propose a novel framework, UMEval, for knowledge-augmented and explainable assessments of the semantic similarity and relatedness of medical terms using large language models (LLMs).

UMEval first performs knowledge retrieval and enrichment based on the Unified Medical Language System (UMLS) and authoritative sources, while adopting a noise-aware selection strategy to quantify and control the uncertainty introduced by candidate definitions and semantic paths. Then, it utilizes LLMs to generate similarity and relatedness scores with natural language explanations, which are verified by a supervisor for factual alignment, and finally produces structured output that includes the score, explanation, supporting references, and supervision logs. By integrating model predictions with verifiable medical knowledge and providing an audit trail, UMEval ensures the interpretability, reliability, and clinical trustworthiness of results.

We conduct extensive experiments on six benchmark datasets against 13 representative baselines. The results indicate that UMEval generally improves over state-of-the-art baselines, including GPT-5.2 and BioLORD, across most settings. Notably, the best-performing configurations reach an agreement with expert ratings of up to 0.88, and the largest observed relative improvement attains up to 21.80.

UMEval offers a reliable and explainable solution for medical term semantic evaluation. The enhanced alignment with expert annotations and transparent reasoning highlights its potential for clinical terminology normalization, decision support, and interoperability in health information systems.