Enhancing semantical text understanding with fine-tuned large language models: A case study on Quora Question Pair...

Area of Science:

• Natural Language Processing (NLP)
• Machine Learning
• Artificial Intelligence

Background:

• Semantic text understanding is crucial in NLP, with datasets like Quora Question Pairs (QQP) used for evaluation.
• A previous study developed a Siamese Convolutional Neural Network (S-CNN) achieving an 82.02% F1 score for text similarity.

Purpose of the Study:

• To explore the effectiveness of large language models (LLMs) in text similarity tasks.
• To compare various fine-tuning approaches for LLMs, including prompt engineering, n-shot learning, and supervised learning with Low-Rank Adaptation (LoRA).

Main Methods:

• Leveraged 5 pretrained LLMs and compared their performance using F1 score.
• Employed 10-fold cross-validation for supervised model training and evaluation, mirroring a previous study's design and dataset.
• Conducted a secondary study comparing 7B and 70B parameter LLMs using the GLUE benchmark after fine-tuning with a corpus.

Main Results:

• The fine-tuned LLaMA 7B model (qLLaMA_LoRA-7B) achieved the highest F1 score of 84.9% on the QQP dataset, outperforming Alpaca_LoRA-65B and the S-CNN model.
• The fine-tuned LLaMA3.1-70B model (qLLaMA3.1_LoRA-70B) achieved an F1 score of 74.4% on the GLUE benchmark, outperforming the 7B model (71.9%).
• Task-specific supervised fine-tuning improved LLM performance compared to larger pretrained models without fine-tuning or with n-shot learning.

Conclusions:

• An effective LLM fine-tuning framework was demonstrated, highlighting the importance of fine-tuning for improved performance.
• Supervised fine-tuning of LLMs yields better results than using larger models without fine-tuning or with n-shot learning.
• Fine-tuning larger LLMs further enhances performance compared to smaller ones, with potential applications in various document similarity tasks.