ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understanding

Area of Science:

• Radiology and Artificial Intelligence
• Medical Imaging Analysis
• Natural Language Processing in Healthcare

Background:

• Existing visual question answering (VQA) benchmarks for chest radiology often rely on template-based queries.
• There is a need for a comprehensive benchmark that reflects diverse and clinically authentic radiological reasoning skills.

Purpose of the Study:

• To introduce ReXVQA, the largest benchmark for VQA in chest radiology.
• To evaluate the performance of state-of-the-art multimodal large language models (LLMs) on clinically relevant radiological tasks.
• To compare AI performance against human expert radiologists.

Main Methods:

• Developed ReXVQA with 694,481 questions and 160,000 chest X-ray studies.
• Included five core radiological reasoning skills: presence assessment, location analysis, negation detection, differential diagnosis, and geometric reasoning.
• Evaluated eight SOTA LLMs and conducted a human reader study with 3 senior radiology residents.

Main Results:

• The best model, MedGemma, achieved 83.24% overall accuracy on the ReXVQA benchmark.
• MedGemma outperformed the best human reader (77.27% accuracy) in a reader study.
• AI models showed distinct performance patterns compared to human readers, with variable inter-reader agreement.

Conclusions:

• ReXVQA establishes a new standard for evaluating generalist radiological AI systems.
• AI performance now exceeds human evaluation on chest X-ray interpretation tasks, a significant milestone.
• The benchmark facilitates the development of next-generation AI systems for expert-level clinical reasoning.