LLM-IE: a python package for biomedical generative information extraction with large language models.

Area of Science:

• Biomedical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show promise for biomedical information extraction (IE).
• Existing challenges in prompt engineering and algorithm development limit LLM application in IE.
• There is a lack of dedicated software for creating comprehensive IE pipelines.

Purpose of the Study:

• To develop a user-friendly Python package, LLM-IE, for constructing end-to-end biomedical information extraction pipelines.
• To address the persistent challenges in prompt engineering and algorithm design for LLM-based IE.
• To provide essential building blocks for robust and efficient IE system development.

Main Methods:

• Developed LLM-IE, a Python package supporting named entity recognition, entity attribute extraction, and relation extraction.
• Implemented an interactive LLM agent for schema definition and prompt design.
• Utilized state-of-the-art prompting algorithms and visualization features.
• Benchmarked LLM-IE performance on the i2b2 clinical datasets.

Main Results:

• The sentence-based prompting algorithm achieved over 70% strict F1 for entity extraction in an 8-shot setting.
• The system demonstrated approximately 60% F1 for entity attribute extraction.
• LLM-IE successfully supports key IE tasks including NER, entity attribute extraction, and relation extraction.

Conclusions:

• LLM-IE provides a foundational toolkit for developing advanced biomedical information extraction pipelines.
• The package facilitates schema definition, prompt design, and utilizes effective prompting algorithms.
• Future work will focus on expanding LLM-IE's capabilities and enhancing its computational efficiency.