Tap&Say: Touch Location-Informed Large Language Model for Multimodal Text Correction on Smartphones.

Area of Science:

• Human-Computer Interaction
• Natural Language Processing
• Machine Learning

Background:

• Mobile text editing via voice input faces challenges in differentiating commands from dictation and identifying edit locations.
• Existing multimodal systems struggle to effectively integrate touch and voice for accurate text correction.

Purpose of the Study:

• To introduce Tap&Say, a novel multimodal system designed to enhance text correction accuracy on mobile devices by integrating touch interactions with Large Language Models (LLMs).
• To address the challenges of distinguishing editing commands from dictation and pinpointing edit locations in voice-based text editing.

Main Methods:

• Developed Tap&Say, a system combining touch input for intent and location with voice input for corrections.
• Proposed a novel 'touch location-informed attention' layer to integrate tap coordinates into the LLM's attention mechanism.
• Fine-tuned the LLM on synthetic data including touch locations and correction commands.

Main Results:

• The touch location-informed LLM achieved significantly higher text correction accuracy compared to the state-of-the-art VT method.
• A user study showed Tap&Say reduced task completion time by 16.4% and keyboard clicks by 47.5% compared to VT.
• Users expressed a preference for the Tap&Say system over existing methods.

Conclusions:

• Tap&Say effectively resolves key challenges in voice-based mobile text editing by leveraging multimodal input.
• The proposed touch location-informed attention mechanism is crucial for improving LLM-based text correction.
• Tap&Say offers a more efficient and user-preferred solution for mobile text correction.