Extracting Knowledge From Scientific Texts on Patient-Derived Cancer Models Using Large Language Models: Algorithm...

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Oncology
• Computational Biology

Background:

• Patient-derived cancer models (PDCMs) are crucial for cancer research and preclinical studies.
• The volume of PDCM-related publications has surged, necessitating efficient knowledge extraction.
• Large language models (LLMs) offer advanced capabilities for processing scientific literature at scale.

Purpose of the Study:

• To investigate LLM-based systems for automated extraction of PDCM-related entities.
• To compare direct prompting and soft prompting techniques for entity extraction.

Main Methods:

• Explored direct prompting (manual prompt design) and soft prompting (trainable continuous vectors).
• Evaluated both approaches across proprietary (GPT4-o) and open (LLaMA3) LLMs.
• Utilized a manually annotated dataset of 100 PDCM abstracts with 15 entity types.

Main Results:

• GPT4-o with direct prompting achieved F1-scores of 50.48 (exact match) and 71.36 (overlapping match).
• LLaMA3 soft prompting significantly improved performance over direct prompting (exact match: 7.06 to 46.68; overlapping match: 12.0 to 71.80).
• LLaMA3 soft prompting slightly outperformed GPT4-o direct prompting in the overlapping match setting.

Conclusions:

• Soft prompting enhances the performance of smaller open LLMs for PDCM entity extraction.
• Training soft prompts on open models can yield performance comparable to proprietary LLMs.
• This approach facilitates scalable knowledge discovery in PDCM research.