UMEval: a unified framework for explainable medical term semantic evaluation with large language models.

Area of Science:

• Medical Informatics
• Natural Language Processing
• Artificial Intelligence in Healthcare

Background:

• Accurate medical term semantic evaluation is crucial for patient safety, diagnosis, and healthcare interoperability.
• Current methods lack comprehensive knowledge, interpretability, and reliability.
• There is a need for advanced frameworks to address these limitations.

Purpose of the Study:

• To introduce UMEval, a novel framework for knowledge-augmented and explainable medical term semantic evaluation.
• To leverage large language models (LLMs) for assessing semantic similarity and relatedness of medical terms.
• To enhance the interpretability and reliability of medical term evaluation.

Main Methods:

• Knowledge retrieval and enrichment using Unified Medical Language System (UMLS) and authoritative sources.
• Noise-aware selection strategy to manage uncertainty in definitions and semantic paths.
• LLM-based score generation with natural language explanations, verified by a supervisor.

Main Results:

• UMEval outperforms 13 state-of-the-art baselines, including GPT-5.2 and BioLORD.
• Achieved agreement with expert ratings up to 0.88.
• Demonstrated a maximum relative improvement of 21.80%.

Conclusions:

• UMEval provides a reliable and explainable solution for medical term semantic evaluation.
• The framework enhances alignment with expert annotations and offers transparent reasoning.
• Potential applications include clinical terminology normalization, decision support, and health information system interoperability.