Evaluation of large language models in female malignancy Q&A

Area of Science:

• Artificial Intelligence in Medicine
• Oncology
• Natural Language Processing

Background:

• Foundational large language models (LLMs) are crucial for AI in medicine.
• Limited evaluation studies exist on LLM Q&A performance in female malignancy.

Purpose of the Study:

• To evaluate and compare the Q&A capabilities of popular LLMs in female malignancy.
• To establish a benchmark dataset for assessing LLM performance in this specialized domain.

Main Methods:

• A cross-sectional study was conducted using a benchmark dataset of 205 female malignancy Q&A.
• Seven popular LLMs were evaluated against three human oncologists.
• Performance metrics included accuracy, consistency, latency, and inter-model agreement.

Main Results:

• Llama-3.1-405B, OpenAI o1, and DeepSeek-R1 exceeded 80% accuracy and consistency.
• Llama-3.1-405B showed higher accuracy and lower latency than OpenAI o1.
• Removing token limits improved OpenAI o1 and DeepSeek-R1 accuracy significantly, nearing Llama-3.1-405B's performance.

Conclusions:

• Llama-3.1-405B is a strong baseline for female malignancy Q&A.
• OpenAI o1 and DeepSeek-R1 show promise, especially without token limits, but require consistency improvements.
• LLMs need further refinement with clinical data to overcome challenges in complex medical terminology and scenarios.