A deep learning framework for real-time prediction of the behavioral state transition during predation

The shift from searching for prey to pursuing it is a critical behavioral transition. Previous studies have identified this transition through post hoc analyses using parameters such as bearing angles and velocities. However, the inability to detect this behavioral shift in real time has hindered attempts to validate brain functions with temporal precision during the shift. To address this issue, we conducted experiments with laboratory mice and developed a deep learning-based method to classify and predict the search-to-pursuit transition. Results showed that the latency period from prey appearance to pursuit correlated with success rates and decreased with experience, suggesting the importance of this transition in predatory behavior. To utilize this transition-identification method in real experiments, we developed a deep learning framework that can detect and predict the transition in real time using video streams. A lightweight YOLOv11n detector enhanced with Ghost modules extracts the kinematic features of the mouse and prey. A spatiotemporal network (STNet) then processes these features. STNet combines GRU-based temporal encoding, attention, and residual convolutional modules for dual-task learning, including behavioral state recognition and prospective transition prediction. STNet achieves high classification accuracy (0.916), strong predictive performance (area under the curve [AUC] = 0.881), and reliable calibration (expected calibration error [ECE] = 0.133; Brier score = 0.082). STNet successfully predicts transitions up to 0.80 s in advance and generalizes robustly across individuals and task difficulty. This open-source approach provides a real-time, video-stream-based method for identifying temporally specific behavioral transitions. This capability enables precisely-timed, closed-loop manipulations to dissect the neural circuits of decision-making.