Large Language Model-Based Classification of Case Report Abstracts: A Pilot Study on Interactions Between...

The growing volume of biomedical literature, especially in oncology, necessitates automated tools for extracting clinically relevant information. Large language models (LLMs) offer promising capabilities for data extraction. However, their potential to extract clinically relevant information from case reports detailing rare treatment interactions remains underexplored.

We systematically searched PubMed for case reports on interactions between radiotherapy (RT) and pembrolizumab, cetuximab, or cisplatin. A random sample of 100 report abstracts for each therapy was manually classified by two independent medical experts using 23 Boolean questions about patient demographics, treatment, cancer type, and outcomes with mutually exclusive answers, forming a ground truth. An LLM-based system with the open-source Generative Pretrained Transformer (GPT) models (GPT-OSS-120B and GPT-OSS-20B) was applied to classify these reports and the remaining data set entries using the defined question structure. Performance of the approach was evaluated using the standard classification metrics accuracy, precision, recall, and F1-scores.

The searches yielded 320 (pembrolizumab), 147 (cetuximab), and 2055 (cisplatin) publications. Inter-rater agreement for manual classification was high (Cohen's kappa = 0.85), though lower for specific outcome and cancer type questions. The LLM-based classification (GPT-OSS-120B model) achieved high overall performance with an F1-score of 93.64% (95.19% accuracy, 93.23% precision, 94.05% recall). Performance was consistent across systemic therapies (STs), with the GPT-OSS-20B model showing similar results (F1-score 93.22%). Analysis of the entire data sets revealed that 56.14% of publications described patients who received both RT and ST. Proportions of positive and negative outcomes varied by therapy and sequencing.

LLM-based classification systems demonstrate high performance for curating scientific case reports on RT and ST interactions. These findings support their potential for high-throughput hypothesis generation and knowledge base construction, particularly for underutilized case reports, with even smaller open-source models proving to be effective.