Automated standardization and harmonization of laboratory units in large-scale clinical data using open-source R...

Area of Science:

• Biomedical Informatics
• Clinical Data Science
• Health Data Standardization

Background:

• Secondary use of clinical laboratory data is hindered by inconsistent unit representations in electronic medical records.
• Heterogeneous multi-source databases suffer from non-standard unit strings and fragmented mappings, limiting interoperability and introducing bias.
• Existing standards like UCUM and LOINC face challenges in real-world data application.

Purpose of the Study:

• To develop and evaluate open-source functions for automated standardization of laboratory units to UCUM-valid formats.
• To harmonize laboratory results to reference units within LOINC groups for improved data consistency.
• To enhance the secondary use and interoperability of large-scale clinical laboratory data.

Main Methods:

• Developed two scalable open-source R functions for unit standardization and value harmonization.
• Standardization employed preprocessing, deterministic mapping, and rule-based correction for heterogeneous unit expressions.
• Applied functions to 163.9 million LOINC-mapped quantitative results; validation used UCUM web service checks.

Main Results:

• Successfully standardized 96.2% of 163.9 million records, reducing unit heterogeneity by 81% (2,019 to 381 UCUM units).
• Harmonization converted 83.4% of records to reference units within minutes, with 100% concordance to UCUM API outputs.
• Harmonization corrected LOINC property inconsistencies in 2.8 million records and improved distributional characteristics in 23% of LOINC groups.

Conclusions:

• Automated, scalable standardization and harmonization of laboratory units are feasible and crucial for large multi-source datasets.
• These functions enable reliable data integration for analytics and machine learning, reducing bias from unit variability.
• The open-source implementation facilitates routine ETL pipeline integration, advancing real-world laboratory data reuse.