Hashtag2Action: Data Engineering and Self-Supervised Pre-Training for Action Recognition in Short-Form Videos

Area of Science:

• Computer Vision
• Machine Learning
• Data Science

Background:

• Short-form social media videos are a rich source of behavioral data.
• Weak labels and privacy concerns hinder direct use in action recognition tasks.

Purpose of the Study:

• To develop an end-to-end pipeline (Hashtag2Action or H2A) for creating a large-scale, ethically curated dataset from short-form videos.
• To pre-train and fine-tune a video action recognition model using this dataset.

Main Methods:

• The Hashtag2Action pipeline employs adaptive hashtag mining, metadata filtering, and vision-based frame validation.
• A VideoMAE V2 backbone was pre-trained self-supervisedly on the curated dataset.
• The model was fine-tuned on established action recognition benchmarks: UCF101, HMDB51, Kinetics-400, and Something-Something V2.

Main Results:

• The H2A dataset comprises 283,582 clips across 386 action categories with minimal manual effort.
• The fine-tuned model achieved high top-1 accuracy: 99.1% (UCF101), 86.1% (HMDB51), 85.5% (Kinetics-400), and 74.3% (SSv2).
• Competitive performance was achieved using only 20% of the original VideoMAE V2 pre-training data.

Conclusions:

• Weakly labeled, ethically curated short-form videos can effectively train action recognition models.
• The H2A dataset and pre-trained weights support reproducible research in short-form video understanding.
• This approach reduces the need for extensive manual annotation in video action recognition.