Lost in the Haystack: Smaller Needles are More Difficult for LLMs to Find.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Large language models (LLMs) struggle with needle-in-a-haystack tasks, requiring information retrieval from extensive contexts.
• Prior research identified positional bias and distractor quantity as key performance factors.
• The impact of gold context size on LLM performance remains under-explored.

Purpose of the Study:

• To systematically investigate how gold context length variations affect LLM performance in long-context question answering.
• To quantify the relationship between gold context size and model accuracy.

Main Methods:

• Experiments were conducted varying gold context lengths in long-context question answering tasks.
• Performance was evaluated across three domains: general knowledge, biomedical reasoning, and mathematical reasoning.
• Seven state-of-the-art LLMs of diverse architectures and sizes were utilized.

Main Results:

• LLM performance significantly declines as gold context size decreases.
• Smaller gold contexts consistently degrade performance and increase positional sensitivity.
• This effect was observed across all tested domains and LLMs.

Conclusions:

• Gold context size is a critical, yet overlooked, factor in LLM performance for long-context QA.
• Diminished gold context size poses a substantial challenge for agentic systems needing to synthesize scattered information.
• Findings offer crucial insights for developing more robust and context-aware LLM-driven systems.