An automatic system for recognizing fly courtship patterns via an image processing method

Fruit fly courtship behaviors composed of a series of actions have always been an important model for behavioral research. While most related studies have focused only on total courtship behaviors, specific courtship elements have often been underestimated. Identifying these courtship element details is extremely labor intensive and would largely benefit from an automatic recognition system. To address this issue, in this study, we established a vision-based fly courtship behavior recognition system. The system based on the proposed image processing methods can precisely distinguish body parts such as the head, thorax, and abdomen and automatically recognize specific courtship elements, including orientation, singing, attempted copulation, copulation and tapping, which was not detectable in previous studies. This system, which has high identity tracking accuracy (99.99%) and high behavioral element recognition rates (> 97.35%), can ensure correct identification even when flies completely overlap. Using this newly developed system, we investigated the total courtship time, and proportion, and transition of courtship elements in flies across different ages and found that male flies adjusted their courtship strategy in response to their physical condition. We also identified differences in courtship patterns between males with and without successful copulation. Our study therefore demonstrated how image processing methods can be applied to automatically recognize complex animal behaviors. The newly developed system will largely help us investigate the details of fly courtship in future research.