Token Probabilities to Mitigate Large Language Models Overconfidence in Answering Medical Questions: Quantitative...

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Medical Informatics

Background:

• Chatbots show promise in medicine, passing medical board exams.
• However, their overconfidence in incorrect answers limits clinical use.

Purpose of the Study:

• To compare token probabilities against chatbots' expressed confidence for predicting medical response accuracy.
• To evaluate if token probabilities are superior to self-reported confidence in assessing chatbot performance.

Main Methods:

• Nine large language models (LLMs) responded to 2522 US Medical Licensing Examination questions.
• Model confidence and response token probability were recorded and analyzed.
• Predictive performance was assessed using AUROCs, calibration error, and Brier score.

Main Results:

• Chatbot accuracy varied, with GPT-4o achieving 89% and Phi-3-Mini 56.5%.
• Expressed confidence poorly predicted accuracy (AUROC 0.52-0.68).
• Token probabilities consistently outperformed confidence (AUROC 0.71-0.87), indicating better accuracy prediction.

Conclusions:

• Chatbots struggle with accurate self-assessment of confidence in medical contexts.
• Token probabilities offer a more reliable method for evaluating chatbot response accuracy.
• Clinicians should not rely on chatbot self-rated certainty; token probabilities are a better alternative.