From memorization to generalization: fine-tuning large language models for biomedical term-to-identifier...

Area of Science:

• Biomedical Informatics
• Natural Language Processing
• Computational Biology

Background:

• Biomedical data integration relies on term-to-identifier normalization for computable and interoperable data.
• Large language models (LLMs) excel at clinical text tasks but show lower accuracy in mapping biomedical terms to ontology identifiers.

Purpose of the Study:

• To investigate the roles of memorization and generalization in LLM term-to-code mapping.
• To assess performance across different ontologies (Human Phenotype Ontology, Gene Ontology) and gene naming systems (HGNC).
• To evaluate the impact of model size and fine-tuning on mapping accuracy.

Main Methods:

• Examined term-to-code mapping performance of LLMs across Human Phenotype Ontology (HPO), Gene Ontology (GO), and HGNC gene naming system.
• Assessed performance of multiple base models and after task-specific fine-tuning.
• Analyzed embedding spaces for semantic alignment between terms and identifiers.

Main Results:

• Accuracy increased with model size, with GPT-4o outperforming Llama 3.1 models.
• Fine-tuning enhanced forward mappings, particularly for GO, but showed minimal gains for gene name-to-HGNC identifier mappings.
• Generalization occurred for HGNC gene symbols but not for HPO/GO identifiers; embedding analysis showed semantic alignment for gene names but not for HPO/GO concepts.

Conclusions:

• Fine-tuning success depends on term popularity and lexicalization.
• Popularity influences baseline accuracy and memorization gains, while lexicalization enables generalization.
• Findings offer a framework to predict fine-tuning effectiveness for biomedical term normalization.