Multimodal convolutional neural network-based algorithm for real-time detection and differentiation of malignant and...

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Gastroenterology

Background:

• Deep learning shows promise for biliary tract cancer detection in digital single-operator cholangioscopy (dSOC).
• Existing image-only models have limitations in diagnosing biliary strictures.
• Multimodal approaches integrating clinical metadata can enhance diagnostic capabilities.

Purpose of the Study:

• Develop a multimodal convolutional neural network (CNN) for detection (CADe) and diagnosis (CADx) of biliary tract cancer.
• Discriminate between malignant, inflammatory, and normal biliary tissue using dSOC videos.
• Integrate clinical metadata into the CNN to improve diagnostic performance.

Main Methods:

• A real-time CNN-based algorithm was developed and validated using dSOC videos and images from 111 patients (15,158 frames).
• Both image-only and metadata-integrated models were established.
• Frame-wise and case-based predictions were validated on video sequences, with model embeddings visualized using class activation maps.

Main Results:

• The CADx approach achieved a per-frame AUC of .871, with sensitivity of .809 and specificity of .773.
• The model demonstrated a negative predictive value of .946 for malignancy.
• Case-based diagnosis correctly identified 6 of 8 malignant and all 12 benign cases.

Conclusions:

• The developed CNN algorithm effectively distinguishes malignant from inflammatory bile duct lesions in dSOC videos.
• CNN-based systems show potential for both CADe and CADx in biliary tract cancer.
• Integrating non-image data, such as clinical metadata, enhances CNN performance for biliary stricture assessment.