LLM Powered Text Entry Decoding and Flexible Typing on Smartphones

Area of Science:

• Natural Language Processing
• Human-Computer Interaction

Background:

• Large language models (LLMs) excel at language tasks but are underutilized in keyboard decoding.
• Keyboard decoding translates user inputs like taps and gestures into text.

Purpose of the Study:

• To develop and evaluate a novel LLM-based decoder for keyboard input.
• To introduce and assess the efficacy of a Flexible Typing method combining various input modalities.

Main Methods:

• Fine-tuning the FLAN-T5 model for keyboard decoding tasks.
• Evaluating performance on user-drawn gestures and real-world tap typing data.
• Conducting a user study to assess the Flexible Typing method.

Main Results:

• The FLAN-T5 decoder achieved 93.1% accuracy on gestures and 95.4% on tap typing.
• Flexible Typing utilized word gestures (35.9%), taps (29.0%), multi-stroke gestures (6.1%), and tap-gestures (29.0%).
• The LLM-based decoder surpassed existing gesture decoders in accuracy.

Conclusions:

• LLM-based decoders offer superior accuracy for keyboard input.
• Flexible Typing enhances user experience and accommodates diverse input preferences.
• This approach advances the field of human-computer interaction for text input.