Application of a General Large Language Model-Based Classification System to Retrieve Information about Oncological...

Area of Science:

• Artificial Intelligence in Medicine
• Computational Biology
• Medical Informatics

Background:

• The exponential growth of oncology publications necessitates automated methods for classifying clinical trials and medical literature.
• Large language models (LLMs) offer potential solutions for automating diverse classification tasks, including information retrieval for oncological trials.

Purpose of the Study:

• To develop and evaluate a general text classification framework utilizing LLMs for oncology clinical trial categorization.
• To assess the performance of various LLMs in classifying oncological trial data.

Main Methods:

• A flexible text classification framework with adaptable prompts, models, and categories was developed.
• The framework was tested on nine binary classification questions across four datasets related to oncological trials.
• Evaluations used a local Mixtral-8x7B-Instruct-GPTQ model and cloud-based LLMs: Mixtral-8x7B-Instruct, Llama3.1-70B-Instruct, and Qwen-2.5-72B.

Main Results:

• The framework achieved high response validity rates (up to 99.88%) across tested LLMs.
• Overall performance metrics included accuracy >94%, precision >92%, recall >90%, and F1-score >92%.
• Question-specific accuracy varied by model but generally remained high, exceeding 77% across all tasks and models.

Conclusions:

• The LLM-based framework demonstrates robust accuracy and adaptability for classifying oncological trials and literature.
• Challenges include prompt dependency and significant computational resource requirements.
• LLMs are poised to play a significant role in automating the classification of oncology research as the technology matures.