Visual Self-Refinement for Autoregressive Models

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Autoregressive models are effective for sequential data, including vision-language tasks.
• Challenges exist in modeling spatial visual data within sequential prediction frameworks.
• Existing methods may suffer from suboptimal results due to the conflict between spatial and sequential data characteristics.

Purpose of the Study:

• To propose a plug-and-play refinement module to enhance spatial correspondence modeling in autoregressive vision-language models.
• To improve the quality and semantic consistency of generated visual sequences.
• To mitigate error accumulation issues inherent in sequential generation.

Main Methods:

• A novel refinement module is introduced as a post-pretraining step.
• The module jointly refines all generated tokens within the autoregressive model.
• It leverages global context and inter-token relationships for improved modeling.

Main Results:

• The proposed method significantly enhances vision-language modeling capabilities.
• It improves the quality of generated visual sequences.
• Error accumulation in sequential generation is effectively mitigated.

Conclusions:

• The refinement module offers a practical solution for improving autoregressive vision-language models.
• The approach successfully addresses the challenge of spatial-sequential data integration.
• The method leads to more semantically consistent and higher-quality outputs in vision-language generation.