Budget-Aware Routing for Long Clinical Text

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computational Linguistics

Background:

• Large language models (LLMs) face challenges with high token costs and deployment expenses.
• Clinical data is often lengthy, varied, and contains redundancy.
• Downstream tasks require concise, high-stakes outputs.

Purpose of the Study:

• To investigate budgeted context selection for LLMs to meet cost and latency constraints.
• To optimize the selection of document subsets within strict token budgets.
• To develop strategies for efficient clinical data processing by LLMs.

Main Methods:

• Framing the problem as a knapsack-constrained subset selection.
• Proposing RCD, a monotone submodular objective balancing relevance, coverage, and diversity.
• Comparing unitization methods (sentence, section, window, cluster) and a routing heuristic.
• Evaluating strategies on MIMIC discharge notes, Cochrane abstracts, and L-Eval datasets.

Main Results:

• Optimal selection strategies vary based on the evaluation setting and budget.
• Positional heuristics excel at low budgets for extractive tasks.
• Diversity-aware methods like MMR enhance LLM generation.
• Selector choice significantly impacts performance more than unitization.
• ROUGE scores saturate for LLM summaries, while BERTScore better indicates quality.

Conclusions:

• Budgeted context selection is vital for efficient LLM deployment in resource-constrained scenarios.
• The RCD objective provides a balanced approach to relevance, coverage, and diversity.
• Task-specific and budget-aware strategies are necessary for optimal LLM performance.
• BERTScore is a more sensitive metric for evaluating LLM-generated summaries than ROUGE.