MetaHarmonizer: robust biomedical metadata harmonization and a contamination control for inflated LLM performance on...

Area of Science:

• Bioinformatics
• Data Science
• Biomedical Informatics

Background:

• Inconsistent metadata hinders the reuse potential of public biomedical data.
• Existing Large Language Model (LLM) approaches face challenges like non-determinism and hallucinated terms.
• LLM performance on benchmarks may overestimate real-world applicability due to data contamination.

Purpose of the Study:

• To develop a robust, automated metadata harmonization system, MetaHarmonizer.
• To address limitations of current LLM-based methods, ensuring accuracy and reliability.
• To facilitate cross-study integration and improve the FAIRness (Findability, Accessibility, Interoperability, Reusability) of biomedical data.

Main Methods:

• MetaHarmonizer comprises SchemaMapper for attribute alignment and OntologyMapper for value standardization.
• Both modules use a multi-stage cascade, escalating to intensive methods only when necessary.
• All mappings are grounded in controlled vocabularies, and LLMs are used as bounded preprocessing components.

Main Results:

• SchemaMapper achieved high accuracy on the GDC schema-matching benchmark, outperforming existing methods.
• OntologyMapper demonstrated strong performance on EFO benchmarks, surpassing text2term and direct LLM inference.
• Calibrated confidence scores enable effective human-in-the-loop triage, with efficient, local, and deterministic inference.

Conclusions:

• MetaHarmonizer offers a robust, scalable, and efficient solution for biomedical metadata harmonization.
• The system overcomes LLM limitations, providing reliable data integration capabilities.
• The developed evaluation methodology can assess other LLM-augmented bioinformatics benchmarks.