Large Language Model-Based Classification of Case Report Abstracts: A Pilot Study on Interactions Between...

Area of Science:

• Biomedical Informatics
• Oncology
• Natural Language Processing

Background:

• The increasing volume of oncology literature presents challenges for information extraction.
• Automated tools are needed to curate clinically relevant data from case reports, especially rare treatment interactions.
• Large language models (LLMs) show potential for such data extraction tasks.

Purpose of the Study:

• To evaluate the performance of LLM-based systems in extracting clinically relevant information from case reports on radiotherapy (RT) and systemic therapy (ST) interactions.
• To assess the utility of LLMs for curating scientific literature detailing rare treatment interactions in oncology.

Main Methods:

• Systematic PubMed search for case reports on RT interactions with pembrolizumab, cetuximab, or cisplatin.
• Manual classification of 100 abstracts per therapy by two experts to establish ground truth.
• Application of open-source Generative Pretrained Transformer (GPT) models (GPT-OSS-120B and GPT-OSS-20B) for classification.
• Performance evaluation using accuracy, precision, recall, and F1-scores.

Main Results:

• LLM-based classification (GPT-OSS-120B) achieved a high F1-score of 93.64% (95.19% accuracy).
• Performance was consistent across different systemic therapies, with GPT-OSS-20B showing similar results (F1-score 93.22%).
• Over 56% of publications involved patients receiving both RT and ST; outcome proportions varied by therapy and sequencing.

Conclusions:

• LLM-based classification systems demonstrate high performance in curating scientific case reports on RT and ST interactions.
• These systems show potential for high-throughput hypothesis generation and knowledge base construction.
• Even smaller open-source LLMs are effective for this task, highlighting their value for underutilized case reports.