Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction...

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multimodal large language models (MLLMs) show promise for chart question answering (CQA).
• Current CQA datasets and training methods often lack focus on fine-grained visual encodings and practical QA task alignment.
• Existing models are typically trained using methods for natural images, neglecting unique chart characteristics like rich text elements.

Purpose of the Study:

• To address the limitations in current CQA datasets and MLLM training for chart understanding.
• To propose a visualization-referenced instruction tuning approach for enhanced CQA performance.
• To develop a more effective method for training MLLMs on chart data.

Main Methods:

• Developed a novel data engine for filtering, refining, and augmenting CQA datasets.
• Employed LLM-based generation techniques to align data with practical QA tasks and visual encodings.
• Utilized an MLLM training strategy involving an unfrozen vision encoder and mixture-of-resolution adaptation for charts.

Main Results:

• The proposed approach significantly outperforms existing state-of-the-art CQA models on established benchmarks.
• The model achieves superior performance even with a reduced number of training examples.
• Empirical studies confirmed the effectiveness of the visualization-referenced instruction tuning and data enhancement techniques.

Conclusions:

• The visualization-referenced instruction tuning approach effectively enhances MLLMs for CQA tasks.
• Data quality and targeted model adaptation are crucial for improving chart understanding in MLLMs.
• The study provides a new dataset split and valuable insights for future CQA research.