Automating time × frequency annotations of delphinid whistles by adapting a foundational transformer neural network

Automated detection of calls is essential to bioacoustic research where it is routine to collect large data sets that preclude effective human annotation. Automated analysis remains challenging due to variations in calls, varying noise backgrounds, and limited annotation data. The recent emergence of very large foundation models has demonstrated exceptional zero-shot and few-shot generalization capabilities, often matching or surpassing specialized models that are fine-tuned for specific domains. We demonstrate that such models can be adapted to bioacoustic tasks. We adapt segment anything model (SAM), a foundational vision model designed for image segmentation, to detect toothed-whale whistle energy in a time-frequency representation of audio. We use an existing tracking algorithm to generate fine-grained time-frequency annotations from the detected whistle energy. The model, SAM-whistle, leverages SAM's pretrained vision transformer network as an encoder network that is adapted to generate spectrogram embeddings that capture information about whistles. The resultant embeddings are decoded using a convolutional decoder that produces time-frequency masks that indicate the presence of whistle energy. Individual whistle contours are produced from the time-frequency masks by the tracking algorithm. Comprehensive evaluations using a subset of the publicly available DCLDE 2011 analyst-annotated dolphin whistle dataset (7211 training whistles and 860 test whistles from common and bottlenose dolphins were used) demonstrate that SAM-whistle has strong performance, achieving an F1 score of 0.897 (precision 0.920, recall 0.876) and demonstrating consistency across a large range of detection thresholds. The model is effective at retrieving nearly complete whistle contours, producing annotations that typically cover 89% of analyst-annotated contours, and producing an expected 1.15 detections for each whistle, indicating that most whistles are not fragmented into multiple detections. Qualitative analysis further demonstrates the effectiveness of SAM-whistle in capturing long-duration whistles and distinguishing unique acoustic patterns. These results highlight the potential of adapting foundation models for specialized bioacoustic analysis tasks, providing a scalable and efficient solution for the automated detection of wildlife vocalizations.