HunFlair2 in a cross-corpus evaluation of biomedical named entity recognition and normalization tools

Area of Science:

• Biomedical Informatics
• Computational Linguistics

Background:

• Biomedical text mining (BTM) is crucial for extracting insights from the growing life sciences literature.
• Entity recognition and normalization are key BTM steps, but tools often fail when applied to data different from their training sets ('in the wild').

Purpose of the Study:

• To evaluate the real-world performance of BTM tools for entity recognition and normalization across different corpora.
• To determine if reported performance metrics are reliable for downstream applications using out-of-domain data.

Main Methods:

• Conducted a cross-corpus benchmark evaluating five selected BTM systems on three public corpora covering four entity types.
• Systematically applied tools to corpora not used during their training to assess generalization capabilities.

Main Results:

• Cross-corpus performance was significantly lower than in-corpus performance, indicating a generalization gap.
• HunFlair2 and PubTator Central demonstrated the best performance among the evaluated tools, though still showing reduced accuracy.
• Results suggest that BTM tool users should anticipate lower performance in practical applications compared to published benchmarks.

Conclusions:

• Existing BTM tools exhibit limited robustness when applied to diverse biomedical text collections.
• Further research is essential to develop more adaptable and reliable BTM systems for real-world biomedical text analysis.