Identifying task groupings for multi-task learning using pointwise V-usable information.

Area of Science:

• Natural Language Processing
• Machine Learning

Background:

• Fine-tuning Large Language Models (LLMs) is crucial for efficiency and performance.
• Multi-task learning can enhance performance but is sensitive to task grouping, risking negative transfer.
• Identifying optimal task groupings remains a challenge.

Purpose of the Study:

• To propose a novel metric for quantifying task relatedness to guide multi-task learning groupings.
• To evaluate the effectiveness of this metric across diverse NLP datasets.

Main Methods:

• Task relatedness is measured using pointwise V-usable information (PVI), a metric of dataset information content.
• Tasks with statistically similar PVI estimates are grouped for joint learning.
• Experiments were conducted on 15 NLP datasets across general, biomedical, and clinical domains, comparing against single-task models and LLMs like Llama and GPT-4.

Main Results:

• Grouping tasks with similar PVI estimates resulted in joint learners achieving competitive performance.
• These joint learners utilized fewer total parameters compared to other methods.
• Consistent performance was observed across different domains.

Conclusions:

• The PVI-based task grouping metric offers a beneficial approach for multi-task learning, especially for domain-specific applications.
• Fine-tuned models remain a strong option, and this metric can enhance their fine-tuning strategies.
• The proposed metric can be integrated into broader fine-tuning methodologies for LLMs.