TextAidAR: Text-Driven Domain Knowledge for Fine-Grained Medical Activity Recognition

While recent advances in activity recognition have achieved remarkable success, these methods depend on large-scale datasets with extensive annotations. Developing fine-grained activity recognition for domain-specific tasks presents unique challenges, including limited labeled data due to privacy concerns and labor-intensive annotation processes. Distinguishing visually similar activities remains difficult, as fine-grained actions often involve subtle differences (i.e., slight variations in hand positioning and tiny object interactions). In trauma resuscitation, critical procedures such as intravenous (IV) placement and temperature measurement occur within small regions of the scene and in complex, crowded environments, further complicating activity recognition. To address these challenges, we reframed medical activity recognition as an image classification task instead of relying on temporal modeling. We introduce TextAidAR, an activity recognition method that incorporates expert-defined domain knowledge to improve visual representation learning. By leveraging vision-text alignment, our approach enhances spatial feature learning using structured medical descriptions provided by domain experts. We also introduce a lightweight module, the MedSpec adapter, to refine pre-trained text embeddings for medical contexts and propose two key loss functions-class-conditioned alignment loss and MedDistinct loss-to enhance the robustness of visual-text alignment. Our studies show that TextAidAR surpasses existing video-based and image-based methods, achieving state-of-the-art performance with a mAP of 0.72 in detecting 13 fine-grained medical activities in real-world trauma resuscitation. On the NurViD dataset, TextAidAR achieves a 41.2% improvement in overall performance compared to the strongest existing baseline.