eNCApsulate: neural cellular automata for precision diagnosis on capsule endoscopes

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Miniaturized Electronics

Background:

• Wireless capsule endoscopy (WCE) offers a noninvasive method for gastrointestinal imaging, but generates vast data, challenging capsule localization and pathology identification.
• Existing deep learning models for tasks like bleeding detection and depth estimation are often too large for direct implementation on capsule hardware.
• Accurate localization of pathologies and the capsule itself is crucial for effective diagnosis and treatment planning in WCE.

Purpose of the Study:

• To develop and implement lightweight deep learning models, specifically Neural Cellular Automata (NCAs), for on-capsule bleeding segmentation and depth estimation.
• To enable efficient image processing directly on miniaturized hardware within a wireless capsule endoscope.
• To improve the precision of diagnosis and localization capabilities of WCE systems.

Main Methods:

• Trained NCAs on capsule endoscopic images for bleeding segmentation and monocular depth estimation.
• Utilized knowledge distillation, using a large foundation model's outputs as pseudo-ground truth, to train a compact NCA architecture.
• Ported trained NCA models to the ESP32 microcontroller for efficient on-device image processing.

Main Results:

• NCAs demonstrated superior accuracy (Dice score) for segmentation compared to other portable models, with over 100x fewer memory parameters.
• NCA-based depth estimation produced visually convincing results, matching or exceeding the realism and detail of the pseudo-ground truth.
• Runtime optimizations on the ESP32-S3 microcontroller led to a significant acceleration of average inference speed (over 3x).

Conclusions:

• Algorithmic adjustments and distillation successfully enabled the encapsulation of NCA models into microcontrollers suitable for WCE.
• This study presents the first successful implementation of reliable bleeding segmentation and depth estimation directly on a miniaturized WCE device.
• The developed technology paves the way for precise diagnosis and visual odometry-based capsule localization within the gastrointestinal tract.