Related Experiment Videos
Automated standardization of legacy biomedical metadata using an ontology-constrained LLM agent
Josef Hardi1, Martin J O'Connor1, Marcos Martínez-Romero1
1Division of Computational Medicine, Stanford University, 3180 Porter Dr, Palo Alto, CA 94304, USA.
Objective:
Descriptive scientific metadata in public repositories are often incomplete and inconsistent with community standards and ontologies, limiting data FAIRness. Large language models (LLMs) offer a promising approach to automatically standardizing such metadata when the LLM is provided with knowledge of the relevant standards in machine-actionable form (e.g., as metadata templates from systems such as the CEDAR Workbench). Entry of metadata standards through prompt engineering offers only fixed snapshots of the relevant standards, however, and it relies on an LLM's pretrained knowledge to interpret and satisfy constraints related to the standard. We evaluate whether giving an LLM access to metadata specifications and authoritative terminology at runtime improves automated metadata standardization.
Methods:
We present ARMS, a tool-augmented LLM agent that retrieves complete CEDAR metadata templates and that queries authoritative biomedical terminology services dynamically at execution time. We compared ARMS with a prompt-based approach on 839 legacy metadata records describing experiment datasets from the Human BioMolecular Atlas Program (HuBMAP) with expert-standardized records serving as the reference standard.
Results:
Across the corpus, ARMS outperformed the prompt-based approach in converting legacy metadata to standards-compliant form, increasing precision from 0.56 to 0.93 and recall from 0.51 to 0.85, with improvements across all field categories and assay types. The largest gains occurred for ontology-constrained fields, where precision increased from 0.36 to 0.92.
Conclusion:
Our findings show that access to appropriate metadata standards can substantially improve LLM-based standardization of legacy metadata. ARMS improves metadata standardization by giving LLMs runtime access to the authoritative resources that define valid metadata. Representation of metadata standards in machine-actionable form enhances the task of rectifying legacy metadata using LLM technology, especially when those standards can be queried dynamically.
