RGB-D scene analysis in the NICU

Area of Science:

• Computer Vision
• Medical Informatics
• Neonatal Care

Background:

• Continuity of care in Neonatal Intensive Care Units (NICUs) relies on meticulous documentation of all clinical events.
• Manual documentation is time-consuming and prone to errors, impacting patient care continuity.

Purpose of the Study:

• To develop an automated scene recognition algorithm for identifying key features in NICU patient environments from images.
• To create a rule-based sentence generator for automatically captioning identified scenes, aiding documentation.

Main Methods:

• Utilized RGB-D camera data from 29 newborn patients, including manual bedside event annotations.
• Implemented image processing for lighting condition classification (brightness, phototherapy).
• Developed a deep neural network leveraging transfer learning for image classification tasks (intervention, bed occupancy, patient coverage), comparing RGB-D fusion techniques.

Main Results:

• Achieved >84% sensitivity and >73% F1 measure across context variables, demonstrating robustness despite class imbalance.
• RGB-D based models outperformed RGB models in most classification tasks.
• Optimal performance was observed with 4-channel image fusion and network fusion at the 11th layer of the VGG-16 architecture.

Conclusions:

• Multimodal computer vision, specifically RGB-D based scene recognition, can effectively automate key aspects of NICU documentation.
• The proposed algorithm and captioning system offer a foundation for a semi-automated charting system to support clinical staff.
• Further development in scene understanding can significantly enhance efficiency and accuracy in neonatal care documentation.