Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second

Area of Science:

• Biophysics
• Computational Imaging
• Microscopy

Background:

• Studying freely moving model organisms requires high-resolution 3D imaging across large fields of view.
• Existing techniques struggle to achieve high speed, resolution, and large FOV simultaneously, limiting behavioral studies.
• There is a need for advanced imaging systems capable of capturing dynamic 3D behavior at scale.

Approach:

• Introduced 3D-RAPID, a computational microscope utilizing a synchronized 54-camera array for high-speed 3D topographic video capture.
• Developed a self-supervised 3D reconstruction algorithm using a CNN, trained with stereo overlap and ray-propagation physics, to fuse images into a 3D height map.
• Achieved high spatiotemporal throughput (230 fps, >5 GP/s) over a 135 cm^2 area.

Key Points:

• 3D-RAPID enables simultaneous 3D reconstruction and seamless fusion of 54 images into a globally consistent composite with a coregistered 3D height map.
• The self-supervised CNN-based reconstruction algorithm is robust and scalable for long videos and varying camera array sizes.
• Demonstrated parallelized 3D observation of ants, fruit flies, and zebrafish larvae at high spatiotemporal resolution.

Conclusions:

• 3D-RAPID overcomes limitations in existing behavioral imaging techniques, offering unprecedented 3D observation capabilities.
• The scalable hardware and software design facilitates large-scale, high-throughput 3D behavioral analysis of multiple organisms.
• This technology opens new avenues for understanding complex behaviors in model organisms.