Towards Dataset-Scale and Feature-Oriented Evaluation of Text Summarization in Large Language Model Prompts

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Human-Computer Interaction

Background:

• Advancements in Large Language Models (LLMs) and Prompt Engineering have democratized chatbot customization.
• Prompt evaluation at scale remains challenging due to the complexity of assessing numerous test instances.
• Existing methods often rely on traditional quality metrics, which can be insufficient for nuanced prompt assessment.

Purpose of the Study:

• To address the complexity of prompt evaluation for Large Language Models (LLMs).
• To introduce a feature-oriented workflow for systematic prompt evaluation.
• To present Awesum, a visual analytics system designed to aid in prompt refinement for text summarization.

Main Methods:

• Conducted a comprehensive literature review and pilot study to identify challenges in prompt evaluation.
• Developed a feature-oriented workflow using summary characteristics (e.g., complexity, formality) instead of traditional metrics like ROUGE.
• Introduced Awesum, a visual analytics system with a novel Prompt Comparator for interactive prompt refinement.

Main Results:

• The feature-oriented workflow and Awesum system simplify systematic prompt evaluation for non-technical users.
• The system effectively aids in identifying optimal prompt refinements for text summarization.
• Practitioner evaluations confirmed the system's effectiveness and general applicability across domains.

Conclusions:

• The proposed feature-oriented workflow and Awesum system significantly lower the barrier for systematic prompt evaluation.
• The approach shows potential for generalization to other Natural Language Generation (NLG) and image-generation tasks.
• Future research should focus on feature-oriented evaluation and human-agent interaction challenges in LLM prompt engineering.