Consensus Finding Among LLMs to Retrieve Information About Oncological Trials

Area of Science:

• Oncology
• Biomedical Informatics
• Artificial Intelligence

Background:

• Automated classification of medical literature is crucial, particularly in oncology.
• Large language models (LLMs) show promise for classifying biomedical literature and clinical trials.
• Previous research established LLMs' utility in this domain.

Purpose of the Study:

• To evaluate the effectiveness of a consensus-based approach for improving LLM classification performance.
• To determine the extent to which consensus enhances accuracy in oncological trial classification.

Main Methods:

• Three LLMs (Mixtral-8x7B, Meta-Llama-3.1-70B, Qwen2.5-72B) were employed.
• Classification of oncological trials was performed across four datasets using nine distinct questions.
• Performance metrics including accuracy, precision, recall, and F1-score were assessed for individual models and consensus outputs.

Main Results:

• Consensus was reached in 93.93% of classification tasks.
• The consensus approach yielded superior performance metrics: 98.34% accuracy, 97.01% precision, 98.11% recall, and 97.55% F1-score.
• These results significantly outperformed individual LLM performances.

Conclusions:

• A consensus-based LLM framework demonstrates high accuracy and adaptability for classifying oncological trials.
• This approach holds potential for advancing biomedical research and clinical trial management.
• The findings support the integration of consensus mechanisms in AI-driven literature analysis.