A Dataset for Evaluating Contextualized Representation of Biomedical Concepts in Language Models

Area of Science:

• Biomedical Informatics
• Natural Language Processing
• Computational Linguistics

Background:

• Capturing context-dependent semantic representations of biomedical terms is a significant challenge.
• Existing evaluation benchmarks are insufficient for assessing language models' understanding of biomedical concepts in context.

Purpose of the Study:

• To introduce BioWiC, a novel dataset designed to evaluate language models' ability to encode biomedical terms within their specific contexts.
• To establish a benchmark for assessing context-dependent embeddings in biomedical corpora.

Main Methods:

• Development of the BioWiC dataset, comprising 20,156 instances and over 7,400 unique biomedical terms.
• Intrinsic and extrinsic evaluations of the BioWiC dataset.
• Experiments using various discriminative and generative large language models to establish baseline performance.

Main Results:

• BioWiC is the largest Word-in-Context (WiC) dataset for the biomedical domain.
• The dataset is suitable as a reliable benchmark for evaluating context-dependent embeddings in biomedical text.
• Established robust baseline performance for large language models on the BioWiC benchmark.

Conclusions:

• BioWiC provides a crucial resource for advancing natural language understanding in the biomedical field.
• The dataset facilitates the development and evaluation of more sophisticated language models for biomedical applications.
• Future research can build upon the established baselines for improved biomedical NLP performance.