Comparison of pipelines, seq2seq models, and LLMs for rare disease information extraction

Area of Science:

• Biomedical Natural Language Processing (NLP)
• Information Extraction
• Knowledge Discovery

Background:

• End-to-end relation extraction (E2ERE) is crucial for building biomedical knowledge graphs.
• Existing E2ERE methods often struggle with complex entities (discontinuous, overlapping, nested) found in document-level tasks.
• The RareDis dataset presents a challenging use-case for E2ERE due to these complex traits.

Purpose of the Study:

• To evaluate and compare the performance of three main E2ERE paradigms on a complex dataset (RareDis).
• To determine the most suitable E2ERE approach for biomedical NLP tasks, especially when dealing with intricate data structures.
• To provide empirical evidence guiding the choice of E2ERE models in the biomedical domain.

Main Methods:

• Comparative analysis of three E2ERE paradigms: pipeline (NER + RC), sequence-to-sequence, and generative large language models (LLMs).
• Utilized the RareDis dataset, characterized by complex entities, for evaluating representative models from each paradigm.
• Validated findings on a second dataset focused on chemical-protein interactions.

Main Results:

• Pipeline-based E2ERE models demonstrated superior performance on the RareDis dataset.
• Sequence-to-sequence models also performed competitively, closely following pipeline approaches.
• Large language models (LLMs) were found to be more effective in zero-shot scenarios, but less optimal than trained models when data is available.

Conclusions:

• For complex biomedical relation extraction tasks with available training data, traditional pipeline or sequence-to-sequence models are recommended over large language models.
• The study highlights the continued relevance and effectiveness of established E2ERE methods for specialized biomedical NLP challenges.
• This research offers the first E2ERE analysis on the RareDis dataset, contributing valuable insights for the field.