Speech-touch integration for affective human-robot interaction: a scoping review

Artificial intelligence is increasingly capable of expressing empathy through language, yet the integration of physical touch-an important cue for social connection-remains fragmented. Although robots utilise language or touch individually, few systems coordinate both modalities, potentially limiting their capacity for affective human-robot interaction (HRI). This scoping review maps social robots that combine spoken language and tactile interaction (e.g., hugging, stroking, warmth, vibration), examines how these modalities are coordinated in existing systems, and synthesises reported user outcomes and design implications.

Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines, searches across five databases (IEEE Xplore, PubMed, ACM, Web of Science, Scopus) and supplementary web sources identified 11 distinct HRI implementations that pair speech with active or invited touch. Of these, eight implementations included explicit comparison conditions (e.g., speech-only vs. speech + touch, or touch-only vs. touch + speech), enabling assessment of the added value of combining modalities.

Across comparative studies, combining speech and touch showed potential to be more effective than speech-only or touch-only HRI in some contexts. This integration can make robots appear more caring, empathic, and human-like, while strengthening attachment, increasing willingness to self-disclose, and helping users feel calmer (e.g., lower heart rate). However, outcomes were implementation-dependent, with some studies reporting no additional benefit from the combined modalities. Across the evidence base, the review found a consistent suggestive pattern that warm (e.g., near skin temperature), soft, naturalistic touch tends to support more positive affective HRI outcomes than cold, rigid, "mechanical" touch. The evidence base was also largely dominated by short, lab-based studies using existing, typically rigid robotic platforms not purpose-built for affective speech-touch interaction.

Speech-touch integration in social HRI is a small but promising area, particularly for healthcare and emotional-support applications (e.g., supporting children in hospital). Despite this potential, very few robots are purpose-built for coordinated speech and touch. Affective speech-touch HRI remains challenging because of its psychological, socio-cultural, and engineering demands. Progress will likely require soft, safe, warm, and increasingly autonomous systems that move beyond repurposed rigid platforms.

https://doi.org/10.17605/OSF.IO/2PA6J, identifier OSF.IO/2PA6J.