Tap&Say:触摸位置信息的大型语言模型用于智能手机上的多模式文本校正
Maozheng Zhao1, Shanqing Cai2, Shumin Zhai2
1Department of Computer Science Stony Brook University Stony Brook, New York, USA.
Proceedings of the SIGCHI conference on human factors in computing systems. CHI Conference
|December 25, 2025
概括
Tap&Say是一个新的多式联络系统,通过将触摸和语音输入与大型语言模型 (LLM) 结合起来,改进了移动文本校正. 这种方法可以准确地区分命令和指令,并精确定位编辑,增强用户体验.
科学领域:
- 人与计算机的交互
- 自然语言处理自然语言处理.
- 机器学习 机器学习
背景情况:
- 通过语音输入进行移动文本编辑,在区分命令和指令以及识别编辑位置方面面临着挑战.
- 现有的多式联运系统难以有效地整合触摸和语音,以实现准确的文本校正.
研究的目的:
- 推出Tap&Say,这是一种新的多式联络系统,旨在通过与大型语言模型 (LLM) 集成触摸交互来提高移动设备上的文本校正准确性.
- 解决编辑命令与语音编辑的编辑位置的区别和定位编辑命令在基于语音的文字编辑中的挑战.
主要方法:
- 开发了Tap&Say系统,该系统结合了触摸输入来确定意图和位置,并使用语音输入进行校正.
- 提出了一种新的"触摸位置知情注意力"层,将触摸坐标集成到LLM的注意力机制中.
- 对合成数据的LLM进行了微调,包括触摸位置和校正命令.
主要成果:
- 与最先进的VT方法相比,触摸位置信息的LLM实现了明显更高的文本校正准确性.
- 一项用户研究显示,Tap&Say与VT相比,减少了16.4%的任务完成时间和47.5%的键盘点击时间.
- 用户表示他们更喜欢Tap&Say系统,而不是现有的方法.
结论:
- Tap&Say通过利用多式联络输入,有效地解决了基于语音的移动文本编辑方面的关键挑战.
- 拟议的触摸位置告知注意力机制对于改进基于LLM的文本校正至关重要.
- Tap&Say为移动文本校正提供了一个更高效和用户喜欢的解决方案.
相关概念视频
01:14Design Example: Resistive Touchscreen
674
A device engineer plays a crucial role in designing user interfaces for mobile devices. One such interface is the resistive touchscreen, which fundamentally consists of two metallic layers: a flexible upper layer and a rigid lower layer, separated by a narrow gap. The high resistance between these two layers is a key characteristic of this design.
When a user touches the screen, the two layers make contact at a specific point known as the touchpoint. This contact reduces the resistance between...
When a user touches the screen, the two layers make contact at a specific point known as the touchpoint. This contact reduces the resistance between...
674
01:12Types of Errors: Detection and Minimization
9.3K
Error is the deviation of the obtained result from the true, expected value or the estimated central value. Errors are expressed in absolute or relative terms.
Absolute error in a measurement is the numerical difference from the true or central value. Relative error is the ratio between absolute error and the true or central value, expressed as a percentage.
Errors can be classified by source, magnitude, and sign. There are three types of errors: systematic, random, and gross.
Systematic or...
Absolute error in a measurement is the numerical difference from the true or central value. Relative error is the ratio between absolute error and the true or central value, expressed as a percentage.
Errors can be classified by source, magnitude, and sign. There are three types of errors: systematic, random, and gross.
Systematic or...
9.3K
02:07Improving Translational Accuracy
3.5K
3.5K
02:07Improving Translational Accuracy
14.0K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
14.0K
01:00Detection of Gross Error: The Q Test
6.8K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.8K