Facial expression monitoring system for predicting patient's sudden movement during radiotherapy using deep learning

Imaging, breath-holding/gating, and fixation devices have been developed to minimize setup errors so that the prescribed dose can be exactly delivered to the target volume in radiotherapy. Despite these efforts, additional patient monitoring devices have been installed in the treatment room to view patients' whole-body movement. We developed a facial expression recognition system using deep learning with a convolutional neural network (CNN) to predict patients' advanced movement, enhancing the stability of the radiation treatment by giving warning signs to radiation therapists.

Convolutional neural network model and extended Cohn-Kanade datasets with 447 facial expressions of source images for training were used. Additionally, a user interface that can be used in the treatment control room was developed to monitor real-time patient's facial expression in the treatment room, and the entire system was constructed by installing a camera in the treatment room. To predict the possibility of patients' sudden movement, we categorized facial expressions into two groups: (a) uncomfortable expressions and (b) comfortable expressions. We assumed that the warning sign about the sudden movement was given when the uncomfortable expression was recognized.

We have constructed the facial expression monitoring system, and the training and test accuracy were 100% and 85.6%, respectively. In 10 patients, their emotions were recognized based on their comfortable and uncomfortable expressions with 100% detection rate. The detected various emotions were represented by a heatmap and motion prediction accuracy was analyzed for each patient.

We developed a system that monitors the patient's facial expressions and predicts patient's advanced movement during the treatment. It was confirmed that our patient monitoring system can be complementarily used with the existing monitoring system. This system will help in maintaining the initial setup and improving the accuracy of radiotherapy for the patients using deep learning in radiotherapy.