Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization

Area of Science:

• Biomedical informatics
• Natural Language Processing
• Evidence synthesis

Background:

• Systematic reviews synthesize evidence from multiple trials.
• Automated generation of narrative evidence summaries is challenging.
• Current abstractive summarization models lack domain specificity.

Purpose of the Study:

• Evaluate modern neural models for automated biomedical evidence summarization.
• Improve factual accuracy in generated narrative summaries.
• Develop methods for automatic factuality evaluation of evidence syntheses.

Main Methods:

• Abstractive summarization of Cochrane systematic review abstracts.
• Evaluation by medical professionals.
• Development of domain-specific summarization approaches (e.g., key finding demarcation, trial quality emphasis).
• Creation of a new automatic factuality evaluation method using finding directionality inference.

Main Results:

• Summarization systems produce fluent and relevant, but often factually inaccurate, summaries.
• Domain-specific strategies modestly improve factual accuracy.
• Proposed methods show potential for enhancing summary reliability.

Conclusions:

• Automated biomedical evidence summarization requires domain-specific adaptations.
• Factual accuracy remains a key challenge in automated narrative synthesis.
• New evaluation methods are crucial for reliable automated evidence synthesis.