Performance Evaluation of Large Language Models in Cervical Cancer Management Based on a Standardized Questionnaire:...

Area of Science:

• Artificial Intelligence in Medicine
• Computational Health
• Digital Health

Background:

• Cervical cancer is a significant global health challenge, especially in low-resource settings.
• Comprehensive management requires integrated screening, diagnosis, and treatment strategies.
• Large language models (LLMs) offer potential but are underexplored in cervical cancer care.

Purpose of the Study:

• To systematically evaluate the performance and interpretability of LLMs in cervical cancer management.
• To compare the effectiveness of different LLMs based on accuracy and guideline compliance.
• To assess the impact of prompts on LLM performance in a medical context.

Main Methods:

• Selected LLMs from AlpacaEval leaderboard v2.0.
• Developed 100 standardized questions covering cervical cancer screening, diagnosis, and treatment.
• Utilized the CO-STAR framework for prompt engineering.
• Evaluated responses for accuracy, guideline adherence, clarity, and practicality (A-D scoring).
• Employed LIME for interpretability analysis.

Main Results:

• ChatGPT-4.0 Turbo ranked highest (effective rate 94% with prompts) among nine evaluated LLMs.
• Seven models demonstrated stable responses; QiZhenGPT consistently underperformed.
• Prompts improved alignment with human annotations for proprietary models.
• Medical-specialized models showed limited improvement with prompts.

Conclusions:

• Proprietary LLMs (ChatGPT-4.0 Turbo, Claude 2) show promise for clinical decision support in cervical cancer.
• Prompt engineering can enhance LLM accuracy in this domain.
• Further research is needed to integrate LLMs into practical medical applications.