Scalable unit harmonization in medical informatics via Bayesian-optimized retrieval and transformer-based re-ranking

Area of Science:

• Biomedical Informatics
• Data Science
• Health Informatics

Background:

• Clinical datasets often contain inconsistent units, hindering data interoperability and reuse.
• Standardizing units is crucial for accurate analysis and reliable multi-institutional studies.

Purpose of the Study:

• To develop and evaluate a scalable methodology for harmonizing inconsistent units in large-scale clinical datasets.
• To address a key barrier to data interoperability in healthcare.

Main Methods:

• A novel unit harmonization system was designed, combining BM25, sentence embeddings, Bayesian optimization, and a transformer-based classifier.
• A multi-stage pipeline involved filtering, identification, harmonization proposal generation, automated re-ranking, and manual validation.
• The system was evaluated on the Optum Clinformatics Datamart dataset (7.5 billion entries) using Mean Reciprocal Rank (MRR).

Main Results:

• The hybrid retrieval approach (BM25 + sentence embeddings) achieved an MRR of 0.8833, outperforming lexical-only (0.7985) and embedding-only (0.5277) methods.
• A transformer-based reranker further improved performance, achieving a final system MRR of 0.9833.
• The system demonstrated high precision (83.39% at rank 1) and recall (94.66% at rank 5).

Conclusions:

• The hybrid architecture effectively combines lexical and semantic approaches for accurate unit harmonization.
• The framework offers an efficient and scalable solution, reducing manual effort and improving accuracy in clinical data.
• Harmonized data ensures consistency across healthcare systems, enabling seamless reuse in analyses and reliable multi-institutional studies.