ChatIOS: Improving automatic 3-dimensional tooth segmentation via GPT-4V and multimodal pre-training.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Digital Dentistry

Background:

• Accurate 3D tooth segmentation from intraoral scans (IOSs) is crucial for dental treatments like orthodontics and prosthetics.
• Current deep learning methods require enhancement for precise and efficient segmentation of 3D IOS data.

Purpose of the Study:

• To propose and evaluate the ChatIOS framework, integrating GPT-4V and multimodal pre-training for improved 3D tooth segmentation.
• To enhance deep learning algorithms for 3D tooth segmentation in IOS data.

Main Methods:

• Developed the ChatIOS framework using 1800 3D IOS scans from the Teeth3DS dataset.
• Pre-processed 3D IOS data into point clouds and utilized GPT-4V for 2D image descriptions.
• Employed multimodal pre-training with point clouds, 2D images, and text descriptions as input triplets.

Main Results:

• ChatIOS significantly outperformed existing benchmarks like PointNet++ on segmentation quality metrics (mIoU, accuracy, DSC).
• Achieved high segmentation accuracy (e.g., 98.0% for maxillary, 97.9% for mandible).
• Demonstrated efficient processing (approx. 2s per scan) and clinical applicability.

Conclusions:

• The ChatIOS framework enhances the effectiveness and efficiency of 3D tooth segmentation for clinical dental procedures.
• This study pioneers multimodal pre-training for 3D tooth segmentation and explores GPT-4V applications in digital dentistry.