Visual information extraction from documents via classification-guided large vision-language models

Area of Science:

• Computer Vision and Machine Learning
• Document Understanding
• Natural Language Processing

Background:

• Visual Information Extraction (VIE) from visually rich documents is hindered by layout variability and real-world impairments.
• Current VIE methods often require extensive labeled data and layout-specific training, limiting scalability.
• Sequential OCR pipelines and end-to-end models present challenges in accuracy and data requirements.

Purpose of the Study:

• To develop a novel classification-guided large vision-language model (LVLM) framework for multi-type VIE.
• To achieve high accuracy in VIE with minimal supervision and robust zero-shot inference.
• To offer an efficient and scalable solution for complex document understanding in office automation.

Main Methods:

• Proposed a framework that decouples document-type classification from content extraction.
• Employed in-context learning (ICL)-based dynamic prompt engineering for task-specific knowledge injection.
• Utilized a classification-guided LVLM for zero-shot inference across diverse document layouts.

Main Results:

• The zero-shot LVLM achieved an F1-score of 86.43% and a normalized edit distance (NED) of 0.90 on a real-world bidding dataset, outperforming a supervised baseline by 18.35 percentage points in F1.
• Optional domain-specific fine-tuning further boosted performance to 93.65% F1 and 0.93 NED.
• Demonstrated superior robustness against document impairments like seals, watermarks, and low contrast.

Conclusions:

• The proposed classification-guided LVLM framework offers an effective and scalable approach for multi-type VIE.
• Minimal supervision and in-context learning enable robust zero-shot performance across varied document layouts.
• The framework provides a significant advancement for automated document understanding in practical applications.