Multimodal Feature-Based Surface Material Classification

Area of Science:

• Multimodal sensing
• Machine learning for material characterization
• Robotics and human-computer interaction

Background:

• Tool-surface interactions generate rich data, including vibrations, sound, and visual texture.
• Existing surface classification methods often struggle with variations in interaction parameters like force and velocity.
• Robust classification requires integrating diverse sensory inputs and accounting for user variability.

Purpose of the Study:

• To develop and validate a tool-mediated surface classification approach combining vibration, sound, and image data.
• To demonstrate robustness against variable scan-time parameters such as contact force and exploration velocity.
• To achieve accurate classification of textured surfaces using a multimodal feature set.

Main Methods:

• Utilized acceleration sensors, microphones, and camera images to capture tool-surface interactions.
• Extracted perception-related features (hardness, roughness, friction), speech-inspired features (modified cepstral coefficients from acceleration), and image texture features.
• Developed a classification system robust to variations in contact force and exploration velocity without explicit measurement.
• Employed a Naive Bayes classifier with a selected subset of six multimodal features.

Main Results:

• The proposed multimodal approach achieved 74% classification accuracy for textured surfaces.
• The system demonstrated robustness under variable freehand movement conditions and across different users.
• Mitigation of variable contact force and exploration velocity effects was successfully addressed.

Conclusions:

• Combining vibration, sound, and image data enables robust tool-mediated surface classification.
• The developed method offers a practical solution for surface identification in dynamic, unconstrained environments.
• The findings highlight the potential of multimodal sensing for advanced material characterization.