Can small language models handle context-summarized multi-turn customer-service QA? A synthetic data-driven...

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Computational Linguistics

Background:

• Customer-service question answering (QA) systems increasingly use conversational understanding.
• Large Language Models (LLMs) offer high performance but face computational and deployment challenges.
• Small Language Models (SLMs) present an efficient alternative, yet their efficacy in multi-turn, context-aware QA is under-researched.

Purpose of the Study:

• To evaluate instruction-tuned SLMs for context-summarized, multi-turn customer-service QA.
• To assess SLMs' ability to maintain contextual consistency and response quality under computational constraints.
• To investigate a history summarization strategy for preserving conversational state in SLM-based QA.

Main Methods:

• Applied parameter-efficient fine-tuning to adapt SLMs for context-summarized multi-turn QA.
• Developed a synthetic data pipeline for creating a specialized QA dataset.
• Employed a structured evaluation framework with quantitative metrics, human assessments, and LLM-as-a-judge evaluations.
• Conducted a conversation stage-based qualitative analysis.

Main Results:

• Significant performance variation observed among nine evaluated instruction-tuned SLMs.
• Some SLMs achieved performance comparable to commercial LLMs.
• Certain SLMs struggled with dialogue continuity and contextual alignment.
• Parameter-efficient fine-tuning demonstrated effectiveness in adapting SLMs.

Conclusions:

• Instruction-tuned SLMs hold promise for resource-constrained customer-service QA applications.
• Further research is needed to address limitations in dialogue continuity and contextual understanding for some SLMs.
• SLMs offer a viable, efficient alternative to LLMs, contingent on model selection and fine-tuning strategies.