Food Image Segmentation Using Multi-Modal Imaging Sensors with Color and Thermal Data.

Area of Science:

• Dietary Assessment Technology
• Computer Vision
• Sensor Fusion

Background:

• Sensor-based food intake monitoring is rapidly advancing.
• Accurate food image segmentation, especially for similar-looking items, remains a challenge.
• Existing methods often struggle with distinguishing multiple food items in a single image.

Purpose of the Study:

• To develop a novel food image segmentation approach using multi-modal Red-Green-Blue (RGB) and thermal imaging (RGB-T).
• To segment regions of similar-looking food items, including those with varying temperatures.
• To evaluate the effectiveness of combined RGB-T data against individual RGB and thermal data.

Main Methods:

• Utilized a k-means clustering algorithm for multi-modal four-Dimensional (RGB-T) image segmentation.
• Captured RGB and thermal image data for six combinations of two food items each.
• Employed bootstrapped optimization of within-cluster sum of squares (WSS) to determine the optimal number of clusters.

Main Results:

• The combined RGB-T data significantly outperformed individual RGB and thermal data in food image segmentation.
• Achieved a mean F1 score of 0.87 ± 0.1 for RGB-T data.
• RGB data yielded a mean F1 score of 0.66 ± 0.13, and thermal data yielded 0.64 ± 0.39.

Conclusions:

• Multi-modal RGB-T imaging combined with k-means clustering offers a superior solution for segmenting similar-looking food items.
• This approach enhances the accuracy of sensor-based dietary assessment systems.
• Further exploration of RGB-T data fusion holds promise for advancing food recognition and portion size estimation.