Context Variance Evaluation of Pretrained Language Models for Prompt-based Biomedical Knowledge Probing

Area of Science:

• Computational linguistics
• Biomedical informatics
• Artificial intelligence

Background:

• Pretrained language models (PLMs) are increasingly researched for their learned knowledge.
• Fill-in-the-blanks tasks, like cloze tests, are common for knowledge assessment.
• Existing benchmarks like BioLAMA face limitations due to prompt biases and data distribution.

Purpose of the Study:

• To address the unreliability and instability of prompt-based knowledge probing in BioLAMA.
• To develop improved methods for evaluating the factual knowledge of PLMs in the biomedical domain.
• To introduce a novel evaluation metric that better captures PLM understanding beyond simple recall.

Main Methods:

• Introduced context variance in prompt generation to mitigate probing biases.
• Proposed a new rank-change-based evaluation metric to assess PLM knowledge.
• Introduced the concept of 'Misunderstand' to differentiate true understanding from memorization.

Main Results:

• Experiments on 12 PLMs demonstrated that context variance prompts enhance BioLAMA's suitability for large-N-M and rare relations.
• The proposed Understand-Confuse-Misunderstand (UCM) metric proved effective in evaluating PLMs.
• Control experiments helped disentangle genuine understanding from simple data copying.

Conclusions:

• The developed context variance prompts and UCM metric offer a more robust and reliable evaluation of biomedical knowledge in PLMs.
• These advancements address key limitations of previous methods, particularly for complex and long-tailed biomedical data.
• The findings contribute to a deeper understanding of what knowledge PLMs truly acquire and how to effectively measure it.