Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Sensor Fusion

Background:

• Traditional handwriting trajectory reconstruction requires specialized writing surfaces.
• Existing motion-based methods struggle with sensor error accumulation, limiting them to single strokes.

Purpose of the Study:

• To develop a method for reconstructing multi-stroke handwriting trajectories from IMU-enhanced digital pen data.
• To eliminate the need for specialized writing surfaces in trajectory reconstruction.

Main Methods:

• Mapping Inertial Measurement Unit (IMU) data from a digital pen to relative displacement.
• Utilizing pre-processing techniques to synchronize data from pen and tablet at different sampling rates.
• Training a Convolutional Neural Network (CNN) to reconstruct trajectories from raw sensor data.

Main Results:

• Achieved a normalized error rate of 0.176 against tablet ground truth for trajectory reconstruction.
• Demonstrated a character error rate of 19.51% for character recognition from reconstructed trajectories.
• A joint model using IMU and trajectory data improved recognition by 0.75% over sensor-only methods.

Conclusions:

• The proposed CNN-based approach effectively reconstructs multi-stroke handwriting trajectories without segmentation or post-processing.
• The system shows promise for accurate handwriting analysis and character recognition using readily available digital pen technology.