Bio-Inspired Gaze and Neural Command Fusion for Assistive Smartphone Interaction

Area of Science:

• Human-Computer Interaction
• Assistive Technology
• Biomedical Engineering

Background:

• Traditional smartphone interaction methods can be challenging for individuals with certain disabilities.
• Integrating gaze tracking and brain-computer interfaces (BCI) offers a promising avenue for developing more inclusive mobile technologies.

Purpose of the Study:

• To develop and evaluate an assistive smartphone interaction system combining mobile gaze tracking with EEG-BCI commands.
• To assess the system's accuracy, command success rates, and overall usability.

Main Methods:

• An Android application was developed to estimate gaze using the front camera, MediaPipe, and a TFLite model, mapping gaze to screen coordinates.
• Electroencephalography (EEG) commands (Emotiv Cortex) were utilized for click, scroll, and back actions.
• A FastAPI and MongoDB backend managed system data, with Android Accessibility used for action execution.

Main Results:

• The system achieved a high success rate, with 33 out of 36 gaze mappings promoted to active profiles.
• Average static mean error was 390.02 px (p95 error: 789.09 px).
• EEG-BCI command success rates were 89.58% (click), 72.22% (scroll), and 77.78% (back), with an average usability score of 4.21/5.

Conclusions:

• The combined gaze tracking and EEG-BCI system demonstrates effective and usable assistive smartphone interaction.
• The system shows significant potential for improving mobile accessibility for users with motor impairments.
• Further research can optimize gaze estimation accuracy and expand the range of BCI-controlled actions.