Do Multi-Document Summarization Models Synthesize?

Area of Science:

• Natural Language Processing
• Artificial Intelligence

Background:

• Multi-document summarization aims to create concise summaries from multiple sources.
• Accurate synthesis of input information is crucial for applications like summarizing clinical trial results.

Purpose of the Study:

• To evaluate the synthesis capabilities of current multi-document summarization models.
• To identify limitations in how models handle input variations and aggregate information.

Main Methods:

• Experiments were conducted on opinion and evidence synthesis datasets.
• A range of summarization models, including fine-tuned transformers and GPT-4, were tested.
• A novel method involving diverse candidate generation and selection was proposed.

Main Results:

• Existing models demonstrate partial synthesis capabilities but are sensitive to input order and composition.
• The proposed method enhances model synthesis by selecting the best summary aligned with aggregate input measures.
• Models showed imperfect sensitivity to input composition, such as the ratio of positive to negative reviews.

Conclusions:

• Current multi-document summarization models require improvement in synthesizing information accurately.
• The proposed method offers a general and effective approach to enhance synthesis in summarization models.
• Further research can focus on refining model sensitivity to input nuances for more reliable evidence synthesis.