Multi-modal Adversarial Regressive Transformer for Cross-subject Fatigue Detection

Fatigue driving constitutes a factor to traffic accidents, necessitating the importance of fatigue detection. In this paper, we propose leveraging Electroencephalogram (EEG) and Electrooculogram (EOG) data for fatigue detection, developing the Multi-modal Adversarial Regressive Transformer (MART) model. MART encodes and concatenates multiple modalities into an embedded sequence, processed by Transformer focusing on isolated and mixed modalities. Transformer outputs contribute to a regressor for prediction and a domain classifier for domain generalization. MART is flexible to accommodate various modality inputs and allow reducing individual differences with adversarial domain generalization. We test the performance of our proposed method and baselines on the SEED-VIG dataset. This dataset contains 23 subjects, using EEG and EOG features as data and corresponding PERCLOS as labels. Under the subject-dependent setup, our model achieves an RMSE of 0.1015. Under the cross-subject setup, the RMSE of 0.1556 is obtained and decreased to 0.1249 with adversarial domain generalization. This result highlights efficacy and adaptability of MART in fatigue monitoring across subjects.