Automating and Evaluating Large Language Models for Accurate Text Summarization Under Zero-Shot Conditions

Area of Science:

• Natural Language Processing
• Artificial Intelligence

Background:

• Automated text summarization (ATS) is vital for extracting specialized information.
• Zero-shot learning (ZSL) enables large language models (LLMs) to process unseen data.
• LLMs are increasingly used for complex NLP tasks.

Purpose of the Study:

• To evaluate LLM effectiveness in generating accurate summaries under ZSL conditions.
• To explore the impact of retrieval augmented generation (RAG) and prompt engineering on summary accuracy.
• To identify limitations and challenges in applying LLMs to specialized ATS.

Main Methods:

• Combined LLMs with summarization modeling, prompt engineering, and RAG.
• Evaluated summary quality using the METEOR metric.
• Analyzed keyword frequencies via word clouds.

Main Results:

• LLMs demonstrate suitability for ATS tasks under ZSL conditions when augmented with RAG.
• RAG enhances factual accuracy and understanding in LLM-generated summaries.
• Web scraping limitations present a challenge for generalized retrieval.

Conclusions:

• LLMs with RAG show significant potential for specialized ATS.
• Goal misgeneralization and web scraping issues require further investigation.
• Future research should focus on overcoming current limitations for improved ATS performance.