St. Gallen International Breast Cancer Consensus-Based Clinical Decision Validation: Concordance Assessment Between...

Area of Science:

• Artificial Intelligence in Oncology
• Medical Informatics
• Computational Biology

Background:

• The emerging large language model (LLM) DeepSeek shows promise for medical applications.
• Systematic evaluations of DeepSeek's performance in breast cancer clinical decision-making, particularly against expert consensus and leading LLMs, are limited.

Purpose of the Study:

• To assess the concordance of the DeepSeek LLM with international breast cancer expert consensus.
• To compare DeepSeek's performance against Gemini 2.0 Pro and ChatGPT-4o in breast cancer clinical scenarios.

Main Methods:

• 139 consensus questions from the 19th St. Gallen International Breast Cancer Conference (SG-BCC) were analyzed.
• DeepSeek, Gemini 2.0 Pro, and ChatGPT-4o models answered each question five times.
• Concordance rate, answer robustness, and statistical measures (Pearson correlation, absolute proportion difference) were used for comparison.

Main Results:

• DeepSeek-V3 achieved an overall concordance rate of 63.31% with expert consensus.
• DeepSeek-V3 demonstrated high average answer robustness (86.69%) and outperformed Gemini 2.0 Pro and ChatGPT-4o in robustness (p < 0.001).
• DeepSeek-V3's concordance rate for frequent answers was comparable to Gemini 2.0 Pro and ChatGPT-4o (p = 0.849).

Conclusions:

• DeepSeek models exhibit moderate concordance with breast cancer expert panel consensus.
• DeepSeek shows significant advantages in answer robustness, suggesting high potential for clinical decision-making in breast cancer.
• Further research into DeepSeek's application in oncology is warranted.