Action recognition in medical environments for robotic assistance

Area of Science:

• Robotics in Medicine
• Human-Computer Interaction
• Machine Learning for Healthcare

Background:

• Teamwork and seamless collaboration are crucial in medical practice.
• Integrating robotic systems requires smooth human-robot interactions.
• Human action recognition can infer states without explicit input, supporting collaboration.

Purpose of the Study:

• To develop a human action recognition framework for medical handovers.
• To enable robotic assistance by using handovers as implicit cues.
• To facilitate robots replacing the giving party in medical handovers.

Main Methods:

• Utilized skeletal information and machine learning algorithms to derive actions from sequential image data.
• Applied action recognition to datasets from operating room surgeries and patient ward care interventions.
• Abstracted movement patterns using skeletal representation and spatiotemporal information.

Main Results:

• Achieved an F1 score of 0.736 ± 0.045 on the OR dataset using ST-GCN.
• Achieved an F1 score of 0.941 ± 0.009 on the Ward dataset using SkateFormer.
• Observed limitations in distinguishing rapid transition actions and specific OR handover actions.

Conclusions:

• Successfully recognized handover phases in surgical and patient ward settings.
• This framework represents a foundational step towards integrating robotic assistance in medical handovers.
• Enables future robotic systems to interact based on implicit cues during medical procedures.