Fine-grained evaluation of a domain-specific Q&A dataset to support trustworthy medical language models

The effective use of Large Language Models (LLMs) for generating coherent and informative content in specialized domains has largely been driven by the development of robust evaluation strategies. Based on this assumption, we introduce HemoQAL, a domain-specific question-and-answer (Q&A) dataset on hemophilia, derived from recent scientific publications and clinical guidelines. Our main contribution lies in a fine-grained evaluation of the quality of LLM-generated content. First, we carried out a human evaluation in which medical experts assessed the factual accuracy and educational value of the generated Q&A pairs. Second, we conducted a semantic similarity analysis to quantitatively evaluate the alignment between each Q&A pair and its original source material. These lightweight, scalable semantic metrics offer an efficient alternative to more resource-intensive human or LLM-based evaluation pipelines. Our findings show that integrating expert review with semantic similarity measures improves the reliability and trustworthiness of LLM-generated medical content, contributing to the development of dependable AI tools in health informatics.