ChatIOS: Improving automatic 3-dimensional tooth segmentation via GPT-4V and multimodal pre-training

Yongjia Wu ¹, Yun Zhang ¹, Yange Wu ¹

⁰Department of Orthodontics, Stomatology Hospital, School of Stomatology, Zhejiang University School of Medicine, Hangzhou, PR China.

Journal of Dentistry +

|

April 14, 2025

View abstract on PubMed

Summary

The ChatIOS framework enhances 3D tooth segmentation using GPT-4V and multimodal pre-training, improving accuracy and efficiency for dental treatments. This approach pioneers multimodal pre-training for digital dentistry applications.

Area Of Science

Computer Vision
Artificial Intelligence
Digital Dentistry

Background

Accurate 3D tooth segmentation from intraoral scans (IOSs) is crucial for dental treatments like orthodontics and prosthetics.
Current deep learning methods require enhancement for precise and efficient segmentation of 3D IOS data.

Purpose Of The Study

To propose and evaluate the ChatIOS framework, integrating GPT-4V and multimodal pre-training for improved 3D tooth segmentation.
To enhance deep learning algorithms for 3D tooth segmentation in IOS data.

Main Methods

Developed the ChatIOS framework using 1800 3D IOS scans from the Teeth3DS dataset.
Pre-processed 3D IOS data into point clouds and utilized GPT-4V for 2D image descriptions.
Employed multimodal pre-training with point clouds, 2D images, and text descriptions as input triplets.

Main Results

ChatIOS significantly outperformed existing benchmarks like PointNet++ on segmentation quality metrics (mIoU, accuracy, DSC).
Achieved high segmentation accuracy (e.g., 98.0% for maxillary, 97.9% for mandible).
Demonstrated efficient processing (approx. 2s per scan) and clinical applicability.

Conclusions

The ChatIOS framework enhances the effectiveness and efficiency of 3D tooth segmentation for clinical dental procedures.
This study pioneers multimodal pre-training for 3D tooth segmentation and explores GPT-4V applications in digital dentistry.

Keywords:

3-dimensional tooth segmentation Digital dentistry Gpt-4v Intraoral scanners Multimodal pre-training