A deep learning-based multi-modal fusion framework for football foul recognition and referee decision assistance

Accurate referee decision-making remains a critical challenge in football, where the distinction between legal and illegal contact often depends on subtle contextual factors. This paper presents FoulMFNet, an integrated deep learning framework for automated foul recognition designed to assist officiating judgment. The proposed system employs a multi-modal fusion architecture that jointly processes appearance features, motion dynamics, and skeletal representations through dynamically weighted integration. An attention-based temporal focusing mechanism identifies decisive contact moments within extended video sequences, while a confidence calibration module produces well-calibrated probability estimates reflecting prediction reliability. Comprehensive experiments conducted on a dataset comprising 12,463 incident clips from professional matches demonstrate that the proposed framework achieves 87.3% recognition accuracy across five foul categories, outperforming established baseline methods by 4.1 to 10.4 percentage points. Consistency analysis with FIFA-certified referees yields a Cohen's kappa of 0.82, indicating substantial agreement with professional standards. The system maintains real-time performance with median latency of 127 ms, enabling practical deployment alongside existing Video Assistant Referee infrastructure. Interpretability components provide comprehensible visualizations of decision rationale, facilitating referee evaluation of system recommendations.