Automatic Classification of Screen Gaze and Dialogue in Doctor-Patient-Computer Interactions: Computational...

Area of Science:

• Medical Informatics
• Human-Computer Interaction
• Clinical Communication Research

Background:

• Studying doctor-patient-computer interactions is crucial for understanding clinical communication.
• Current methods for data collection are often costly and obtrusive, hindering research.

Purpose of the Study:

• To develop a computational ethnography tool for unobtrusive analysis of doctor-patient-computer interactions.
• To automatically classify combinations of screen gaze and dialogue during clinical consultations.

Main Methods:

• Utilized video data from doctors' internal computer cameras and microphones.
• Employed facial keypoint estimation and voice activity detection to classify interaction types.
• Defined four interaction classes: screen gaze and dialogue, dialogue, screen gaze, and other.

Main Results:

• Achieved an overall classification accuracy of 0.83, comparable to human coders.
• Demonstrated higher accuracy in fully inclusive clinic layouts compared to semi-inclusive ones.

Conclusions:

• The developed classifier offers a non-intrusive method for analyzing screen gaze and dialogue in clinical settings.
• This tool supports researchers, clinicians, designers, and educators in studying doctor-patient-computer interactions.