TIPS: A Framework for Text Summarising with Illustrative Pictures

Area of Science:

• Natural Language Processing
• Computer Vision
• Information Retrieval

Background:

• Generating relevant visual aids for text summaries is challenging.
• Existing methods often rely on single embedding techniques, limiting accuracy.

Purpose of the Study:

• To propose and evaluate the Text-based Illustrative Picture System (TIPS) algorithm.
• To improve the selection of illustrative images for text summaries.

Main Methods:

• TIPS employs a voting process combining results from multiple 'weak' algorithms.
• Text summarization generates sentence-level input for sentence transformers.
• CLIP similarity algorithms assess semantic distances between text and image embeddings.
• A voting mechanism integrates top-scoring image recommendations, down-weighting irrelevant ones.

Main Results:

• Human judges observed a 5-8% increase in highly relevant image matches.
• Judges also noted a 3-7% increase in generally relevant images compared to single-embedding approaches.

Conclusions:

• The TIPS algorithm effectively enhances the relevance of illustrative images for text summaries.
• The multi-algorithm voting approach in TIPS outperforms single-embedding methods.