Enhancing Detection of Message Intents in a Mobile Health Smoking-Cessation Intervention Using Large Language Model...

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Digital Health Interventions

Background:

• Low quit rates persist despite smoking-cessation aids like support groups and nicotine replacement therapy (NRT).
• Mobile health interventions offer accessibility but require timely responses, making accurate intent detection crucial for chatbots.
• Large language models (LLMs) show promise for intent detection but struggle with complex language and imbalanced datasets.

Purpose of the Study:

• To develop and evaluate an AI tool using LLMs for accurate intent detection in smoking-cessation messages.
• To address challenges of dataset imbalance and complexity in user messages from a smoking-cessation support group intervention.

Main Methods:

• Utilized the Llama-3 8B LLM, initially off-the-shelf.
• Fine-tuned the LLM on an annotated dataset with 25 intent categories.
• Implemented data downsampling to mitigate bias from predominant intent categories and explored error correction in human annotations.

Main Results:

• Off-the-shelf LLM performance was poor (F1-scores 0.29-0.41).
• Fine-tuning significantly improved performance (F1-scores 0.72-0.86).
• Fine-tuning combined with downsampling and error correction achieved high F1-scores (0.86-0.90) on a corrected dataset, though performance varied on the full dataset.

Conclusions:

• Domain-specific fine-tuning is essential for LLM performance in smoking cessation.
• Dataset imbalance negatively impacts LLM accuracy, with downsampling offering moderate improvements.
• While error correction can refine annotations, automated downsampling may be a more efficient strategy for improving intent detection.