Out of distribution detection with attention head masking for multimodal document classification

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Detecting out-of-distribution (OOD) data is crucial for reliable AI systems.
• Current OOD detection methods often struggle with multi-modal data and prioritize decision rules over embedding quality.

Purpose of the Study:

• To introduce Attention Head Masking (AHM) for improved OOD detection in both uni-modal and multi-modal settings.
• To enhance the quality of dense embedding representations for better in-distribution and OOD data separation.

Main Methods:

• Applied Attention Head Masking (AHM) to Transformer-based models.
• Evaluated AHM on uni-modal and multi-modal document data.
• Introduced the FinanceDocs dataset for OOD detection research.

Main Results:

• AHM significantly improves embedding quality and data separation.
• Reduced the false positive rate (FPR) by up to 10% compared to state-of-the-art methods.
• Demonstrated effective generalization to multi-modal document data.

Conclusions:

• AHM is an effective technique for enhancing OOD detection in Transformer models.
• The method shows strong performance, particularly for complex multi-modal document analysis.
• The release of FinanceDocs aims to foster further research in document AI and OOD detection.