Evaluating Medical Entity Recognition in Health Care: Entity Model Quantitative Study.

Area of Science:

• Natural Language Processing
• Medical Informatics
• Machine Learning

Background:

• Named Entity Recognition (NER) models are crucial for extracting structured medical information, aiding clinical decision-making and research.
• Advancements in deep learning, including BERT and large language models, have improved NER, but performance varies with medical terminology complexity.
• Existing research often overlooks specific medical context challenges and the impact of lexical factors on NER accuracy.

Purpose of the Study:

• To evaluate various NER models for medical text analysis, focusing on accuracy with complex terminology.
• To investigate the influence of macrofactors on model performance for refining NER models.
• To enhance the reliability of NER models in medical applications.

Main Methods:

• Evaluated 7 NER models (including HMM, CRF, BERT variants, and Gemma) on 3 medical datasets (JNLPBA, BioCreative V CDR, AnatEM).
• Assessed prediction accuracy, resource utilization (CPU/GPU), and hyperparameter tuning impact.
• Screened macrofactors influencing performance using a multilevel factor elimination algorithm.

Main Results:

• Fine-tuned BERT for Biomedical Text Mining achieved top accuracy on Revised JNLPBA (0.932 AVG_MICRO) and AnatEM (0.8494 AVG_MICRO).
• Gemma, with low-rank adaptation, led on BioCreative V CDR (0.9962 AVG_MICRO) but showed variability on other datasets.
• Entity phrase length and number of words per phrase were identified as significant macrofactors affecting model performance.

Conclusions:

• NER models are vital in medical informatics, necessitating optimization through precise data targeting and fine-tuning.
• Findings will improve clinical decision-making and guide the development of more effective medical NER models.