BioGPT: generative pre-trained transformer for biomedical text generation and mining.

Area of Science:

• Biomedical Natural Language Processing
• Artificial Intelligence in Medicine
• Large Language Models

Background:

• Pre-trained language models (PLMs) show promise in the biomedical domain.
• Existing models like BioBERT and PubMedBERT are effective for discriminative tasks but lack generation capabilities.
• The need for generative models in biomedical NLP is growing.

Purpose of the Study:

• Introduce BioGPT, a domain-specific generative Transformer language model.
• Evaluate BioGPT's performance on various biomedical NLP tasks.
• Demonstrate BioGPT's superiority over existing models in both discriminative and generative aspects.

Main Methods:

• Pre-training BioGPT on a large corpus of biomedical literature.
• Evaluating BioGPT on six biomedical NLP tasks, including relation extraction and question answering.
• Conducting a case study on text generation for biomedical terms.

Main Results:

• BioGPT achieved state-of-the-art results on multiple tasks, including BC5CDR, KD-DTI, DDI relation extraction, and PubMedQA.
• Achieved 44.98%, 38.42%, and 40.76% F1 scores on BC5CDR, KD-DTI, and DDI tasks, respectively.
• Obtained 78.2% accuracy on PubMedQA, setting a new benchmark.

Conclusions:

• BioGPT demonstrates strong performance across various biomedical NLP tasks.
• The generative capabilities of BioGPT offer advantages for biomedical text generation.
• BioGPT represents a significant advancement in domain-specific generative language models.