Grammar-constrained decoding for structured information extraction with fine-tuned generative models applied to...

Area of Science:

• Natural Language Processing
• Information Extraction
• Machine Learning

Background:

• Standard encoder-decoder models struggle to guarantee semantic and syntactic constraints in information extraction (IE).
• Constrained decoding approaches, utilizing context-free grammars, offer a way to enforce output structure.
• The effectiveness of domain-specific grammars in guiding IE systems for domain data model compliance remains an open question.

Purpose of the Study:

• To experimentally investigate the impact of grammar-constrained decoding and pointer generators on domain-specific IE performance.
• To evaluate if these techniques improve information extraction results for fine-tuned encoder-decoder models (Longformer, Flan-T5).
• To assess performance in low-resource settings with limited training data (hundreds of examples).

Main Methods:

• Framed the task as slot filling for structured representations from clinical trial abstracts, using the C-TrO ontology.
• Employed Longformer and Flan-T5 encoder-decoder models, comparing performance with and without grammar-constrained decoding and pointer generators.
• Utilized a dataset of 211 annotated clinical trial abstracts for type 2 diabetes and glaucoma.

Main Results:

• Grammar-constrained decoding demonstrated a positive impact, increasing F1 scores by 0.351 (to 0.413) for type 2 diabetes and 0.425 (to 0.47) for glaucoma.
• Pointer generators negatively impacted results, decreasing F1 scores by 0.15 (to 0.263) for type 2 diabetes and 0.198 (to 0.272) for glaucoma.
• Best performance gains from grammar-constrained decoding were observed in low-resource settings.

Conclusions:

• Encoder-decoder models for structure prediction in low-resource IE tasks benefit significantly from grammar-constrained decoding.
• Pointer generator models showed decreased performance, with outcomes dependent on the base model and attention aggregation.
• Further research is needed to structurally investigate how large language model size influences the benefits of grammar-constrained decoding.