Model and Task-Aware Test-Time Scaling Strategies for Large Language and Vision-Language Models in Medicine:...

Area of Science:

• Medical Artificial Intelligence (AI)
• Large Language Models (LLMs)
• Vision-Language Models (VLMs)

Background:

• Test-time scaling enhances LLM and VLM reasoning without retraining.
• Existing scaling methods are established for general domains but underexplored in medical AI.

Purpose of the Study:

• Investigate test-time scaling's impact on medical AI.
• Evaluate scaling across model sizes and task complexities.
• Identify domain-specific bottlenecks and assess robustness against misleading information.

Main Methods:

• Evaluated diverse LLMs and VLMs on medical benchmarks (textual and multimodal).
• Applied three scaling conditions: increased token budgets, sequential scaling, and parallel scaling.
• Tested robustness by introducing misleading clinical authority hints into prompts.

Main Results:

• Reasoning LLMs improved with increased token budgets on complex tasks.
• VLMs showed bottlenecks in visual integration; medically fine-tuned LLMs struggled with calculation tasks.
• Models demonstrated vulnerability to misleading expert hints, despite improved robustness with scaling.

Conclusions:

• General test-time scaling rules require adaptation for medical AI.
• Optimal scaling depends on task complexity: parallel for simple, sequential/larger budgets for complex.
• Safe clinical deployment necessitates addressing VLM integration, reasoning balance, and susceptibility to authority.