Piloting Temperature-Driven Variability in Emergency Diagnostic Accuracy Using a Leading Large Language Model

Area of Science:

• Artificial Intelligence
• Medical Diagnostics
• Clinical Decision Support

Background:

• Large language models (LLMs) utilize a 'temperature' parameter to control output randomness.
• This parameter's influence on clinical diagnostic accuracy, particularly in emergency medicine, is not well understood.
• Understanding temperature's impact is crucial for reliable AI in healthcare.

Purpose of the Study:

• To evaluate the effect of the temperature parameter on GPT-4o's diagnostic accuracy for emergency medicine cases.
• To assess how temperature influences diagnostic divergence and consistency across multiple iterations.
• To determine optimal temperature settings for reliable clinical diagnostic tasks using LLMs.

Main Methods:

• A simulation-based study used four challenging emergency medicine cases.
• GPT-4o generated 10,000 differential diagnoses across five temperature settings (0.0-1.0) and with/without physical exam findings.
• Diagnostic accuracy was benchmarked against gold standards; diagnostic divergence was measured by unique diagnoses generated.

Main Results:

• GPT-4o achieved 100% leading diagnosis accuracy at temperature 0.0, decreasing to 89.4% at temperature 1.0.
• Higher temperatures significantly increased diagnostic inaccuracy and divergence (483% increase from 0.0 to 1.0).
• Case sensitivity to temperature varied, with some diagnoses heavily impacted by physical exam data exclusion.

Conclusions:

• Increasing the temperature parameter in GPT-4o systematically reduces diagnostic accuracy and consistency in emergency medicine scenarios.
• Lower temperature settings (e.g., 0.0) are associated with higher accuracy and reliability, making them potentially preferable for clinical use.
• Transparent reporting of temperature settings is vital for reproducibility in clinical AI research.