EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision
• Natural Language Processing
• Audio Speech Analysis

Background:

• Existing multi-modal AI datasets predominantly use third-person perspectives, unlike human embodied intelligence which relies on first-person sensory input.
• Advancing embodied AI requires datasets that capture the egocentric, first-person viewpoint.

Purpose of the Study:

• Introduce the Egocentric Communications (EgoCom) dataset, a novel resource for embodied AI research.
• Enable advancements in conversational AI, natural language processing, audio analysis, computer vision, and machine learning.

Main Methods:

• Collected 38.5 hours of synchronized, multi-modal data from 34 diverse speakers, capturing egocentric stereo audio and video.
• Annotated data with 240,000 ground-truth, time-stamped word-level transcriptions and speaker labels.
• Developed and evaluated baseline models for two novel applications: conversational turn-taking prediction and multi-speaker transcription.

Main Results:

• Achieved turn-taking prediction within 5% of human performance using Bayesian baselines.
• Improved multi-speaker transcription accuracy by 79% relative to single-perspective methods by combining egocentric data.
• Demonstrated the utility of synchronous, multi-perspective egocentric data for enhancing embodied AI tasks.

Conclusions:

• The EgoCom dataset provides a unique resource for developing more human-like embodied AI systems.
• Egocentric data significantly boosts performance in conversational AI tasks like turn-taking and transcription.
• This work paves the way for more sophisticated AI that understands and interacts with the world from a first-person perspective.