Classifying Clinical Evidence Levels of Cancer Variants in Biomedical Literature Using Machine Learning and Large...

Area of Science:

• Biomedical Informatics
• Computational Biology
• Oncology

Background:

• Automating clinical evidence level classification in biomedical literature is crucial for advancing precision oncology.
• Efficient variant interpretation and informed clinical decision-making rely on accurate evidence categorization.

Purpose of the Study:

• To compare the performance of large language models (LLMs) and machine learning (ML) algorithms in classifying evidence levels for the Clinical Interpretation of Variants in Cancer (CIViC) system.
• To evaluate different prompting strategies for LLMs and feature representations for ML models.

Main Methods:

• Two LLMs (GPT-4.1-mini, Gemini-2.5-Flash) were tested using zero- and few-shot prompting.
• Two ML algorithms (decision tree, XGBoost) were evaluated using Term Frequency-Inverse Document Frequency (TF-IDF) and word embeddings.
• Performance was assessed based on the CIViC evidence level system.

Main Results:

• XGBoost utilizing TF-IDF achieved the highest performance, with a micro-F1 score of 0.83.
• This XGBoost model outperformed both tested LLMs and the decision tree algorithm.
• All models demonstrated better performance on mid-range evidence levels (B-D) compared to high (A) or inferential (E) levels.

Conclusions:

• Current abstract-level evidence classification relies heavily on explicit lexical cues.
• Standalone large language model approaches offer limited additional benefit over traditional ML methods for this task.
• Dataset imbalance and linguistic ambiguity present challenges for classifying extreme evidence levels (A and E).