Learning of Multimodal Representations With Random Walks on the Click Graph

Area of Science:

• Computer Science
• Information Retrieval
• Machine Learning

Background:

• Traditional multimedia retrieval methods often struggle to fully utilize user click data, particularly implicit relationships between text and images.
• Existing approaches using paired or ranked data do not exploit the rich connections within large-scale click logs.

Purpose of the Study:

• To develop a novel model that effectively learns multimodal representations from user click data.
• To improve cross-modal retrieval performance by capturing explicit and implicit relevance within a click graph.

Main Methods:

• Representing user click data as a click graph with images and text queries as vertices.
• Employing a multimodal random walk neural network (MRW-NN) incorporating truncated random walk loss.
• Integrating deep neural network outputs with learned representations.

Main Results:

• MRW-NN successfully learns robust multimodal representations from a large-scale click log dataset (Clickture).
• The model demonstrates superior cross-modal retrieval performance for unseen queries and images compared to state-of-the-art methods.

Conclusions:

• The proposed MRW-NN effectively utilizes implicit connections in click data for enhanced multimedia information retrieval.
• This approach offers a powerful solution for cross-modal retrieval, even with novel or previously unseen data.