Fine-Tuning Large Language Models Using Entity Hallucination Index for Text Summarization.

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Machine Learning

Background:

• Large language models (LLMs) have advanced abstractive summarization.
• Entity-level hallucination (introducing incorrect entities) remains a key challenge in LLM-generated summaries.
• Existing methods struggle to ensure factual accuracy and entity faithfulness.

Purpose of the Study:

• To propose a reward-driven fine-tuning framework to mitigate entity hallucination in abstractive summarization.
• To introduce the Entity Hallucination Index (EHI) as a metric for guiding summarization model fine-tuning.
• To enhance the factuality and robustness of LLM-based summarization.

Main Methods:

• Generated initial summaries using pre-trained LLMs (e.g., Flan-T5, DistilBART, Mistral) on datasets like XSUM.
• Computed the Entity Hallucination Index (EHI) by comparing named entities in generated summaries and gold references.
• Employed reinforcement learning with EHI as the reward signal, using a REINFORCE-style update mechanism for fine-tuning.

Main Results:

• Models fine-tuned with EHI demonstrated significantly lower hallucination rates.
• Informativeness of the summaries was maintained without compromise.
• EHI-guided models exhibited improved generalization on out-of-domain summarization tasks, indicating enhanced robustness.

Conclusions:

• The proposed EHI-guided fine-tuning framework effectively reduces entity hallucination in abstractive summarization.
• This approach offers a practical method for improving the factuality of LLM-generated summaries.
• Accurate entity representation is crucial for reliable abstractive summarization.