Hashtag2Action: Data Engineering and Self-Supervised Pre-Training for Action Recognition in Short-Form Videos

Short-form social-media videos offer a rich, low-cost source of behavioural data, yet their weak or noisy labels and privacy constraints complicate their direct use in action recognition. We introduce an end-to-end pipeline, Hashtag2Action (H2A), that turns raw clips from short-form video platforms (i.e. TikTok) into a large-scale, ethically curated H2A dataset. The pipeline combines adaptive hashtag mining, metadata filtering, and vision-based frame validation to assemble 283,582 clips spanning 386 action categories with minimal manual effort. Using this collection, we pre-train a VideoMAE V2 backbone in a self-supervised manner and fine-tune it on UCF101, HMDB51, Kinetics-400, and Something-Something V2. With only 20% of the original VideoMAE V2 pre-training data, the model achieves 99.1% (UCF101), 86.1% (HMDB51), 85.5% (Kinetics-400), and 74.3% (SSv2) top-1 accuracy. These results show that carefully curated, weakly labelled short-form videos can support competitive downstream performance without additional annotation. To support reproducible research in short-form video understanding, the pre-trained and fine-tuning weights and metadata are publicly available at https://doi.org/10.57967/hf/3179.