Measuring the Quality of CDA to FHIR Transformations Using Text Mining Techniques

Data interoperability remains one of the most pressing challenges in healthcare informatics, limiting the secondary use of health data due to fragmented standards and domain complexity. Transformations between formats, such as from HL7 CDA to FHIR, are complex to develop and difficult to validate. This paper defines three common types of transformation errors: incorrect logical mappings, missing data points, and faulty classifier translations, and presents an algorithm for detecting them. The algorithm assesses transformation quality by measuring the similarity between the input and output of the transformation through a free-text analysis approach, utilizing text mining techniques. It outputs three statistical indicators: schema semantic similarity, equivalent values ratio, and value token count ratio. A prototype was implemented and validated against a structured test suite. The results show that the algorithm reliably detects the presence of these errors. While effective, the sensitivity of the semantic similarity indicator could be improved by integrating a fine-tuned or custom-made semantic text embedding model. Further work includes refining the algorithm, extending its use to transformations between other health data models and investigating applications of the algorithm for other use cases requiring health document similarity assessment.