Sample Size Considerations for Fine-Tuning Large Language Models for Named Entity Recognition Tasks: Methodological...

Area of Science:

• Health Informatics
• Natural Language Processing
• Biomedical Data Science

Background:

• Large language models (LLMs) offer significant potential for health informatics applications.
• However, there is a lack of practical data regarding sample size requirements for fine-tuning LLMs in biomedical and health policy contexts.

Purpose of the Study:

• To evaluate sample size and selection techniques for fine-tuning LLMs.
• To improve named entity recognition (NER) for conflict of interest disclosure statements.

Main Methods:

• Annotated 490 conflict of interest disclosure statements to identify "PERSON" and "ORG" entities.
• Drew 2500 stratified random samples of varying sizes for fine-tuning.
• Trained Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT) models using these samples.
• Assessed the impact of sample size (sentences) and entity density (entities per sentence [EPS]) on NER performance (F1-score) using multiple regression.

Main Results:

• Fine-tuned models achieved high NER performance (F1-score: 0.79–0.96).
• Both sample size and EPS were significant predictors of model performance (P<.001).
• Identified diminishing marginal returns for both sample size (439–527 sentences) and EPS (1.36–1.38).

Conclusions:

• Effective fine-tuning of LLMs for biomedical NER is achievable with modest sample sizes.
• Training data entity density should align with production data.
• Training data quality and model architecture's intended use are critical factors, potentially more so than data volume or model size.