Interactive Natural Language Grounding via Referring Expression Comprehension and Scene Graph Parsing

Area of Science:

• Robotics
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Natural language is key for human-robot interaction.
• Current methods for natural language visual grounding often rely on dialogue systems, leading to cumbersome interactions.
• Ambiguity in natural language queries complicates target object identification.

Purpose of the Study:

• To develop an interactive natural language grounding method without auxiliary information.
• To enable robots to ground complex natural language commands directly to visual targets.
• To improve the efficiency and intuitiveness of human-robot interaction.

Main Methods:

• Proposed a referring expression comprehension network to ground natural language expressions.
• Utilized a visual semantic-aware network to extract visual semantics.
• Employed a language attention network to leverage linguistic context.
• Integrated the comprehension network with scene graph parsing for complex grounding tasks.

Main Results:

• The referring expression comprehension network demonstrated strong performance on three public datasets.
• The interactive natural language grounding architecture was effective in diverse household scenarios.
• The proposed method successfully grounds unrestricted and complicated natural language queries.

Conclusions:

• The developed referring expression comprehension network effectively grounds natural language.
• The interactive natural language grounding architecture enhances human-robot interaction efficiency.
• This approach offers a more direct and less cumbersome way for robots to understand commands.