Related Experiment Videos
Art critic: Multisignal vision and speech interaction system in a gaming context.
IEEE Transactions on Cybernetics
|August 6, 2013
Summary
This study introduces a novel system for real-time game immersion, integrating eye gaze, head pose, and facial expression recognition. This system enhances player engagement by enabling realistic character interactions and dynamic game responses.
Area of Science:
- Human-Computer Interaction
- Computer Vision
- Artificial Intelligence
Background:
- Achieving true player immersion in games requires realistic simulation of the game world.
- Games need to interpret player focus, attitude, gestures, and speech for enhanced realism.
Purpose of the Study:
- To propose a novel system integrating multiple components for real-time game interaction.
- To enhance player immersion through accurate understanding of player cues.
Main Methods:
- Integrated system: eye gaze estimation, head pose estimation, facial expression recognition, speech recognition, and text-to-speech.
- Novel head pose algorithm combining scene flow with a generic head model.
- Facial expression recognition using local binary patterns on the 2-D shape index domain.
Main Results:
- Developed a system capable of real-time player behavior analysis (gaze, head pose, expression, speech).
- Extended system with a Kinect-driven pan-tilt-zoom camera for tracking moving players.
- Demonstrated system effectiveness in a test game, 'Art Critic', showcasing realistic NPC interactions.
Conclusions:
- The integrated system significantly enhances player immersion and interaction realism in gaming.
- The proposed methods for head pose and facial expression recognition show improved performance.
- The 'Art Critic' game serves as a valuable template for future player/NPC interactions.
