通过深度神经网络实现实时摄影现实3D全息
Liang Shi1,2, Beichen Li3,4, Changil Kim3,4
1Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA. liangs@mit.edu.
Nature
|March 11, 2021
概括
这项研究引入了从RGB深度图像中实时生成3D全息图的深度学习管道. 有效的卷积神经网络 (CNN) 能够在消费者硬件和移动设备上实现现实化的全息图.
科学领域:
- 光学和光学
- 计算机视觉
- 人工智能
背景情况:
- 计算机生成全息 (CGH) 对于沉浸式3D显示至关重要,但由于计算密集的模拟,它在焦点控制,遮蔽和实时性能方面面临限制.
- 现有的基于物理的CGH方法在每像素的焦点控制和精确的遮蔽方面存在困难,而弗雷内尔衍射模拟在图像质量和运行时间之间产生了权衡,阻碍了动态应用.
研究的目的:
- 开发基于深度学习的CGH管道,从单个RGB深度图像中实时合成现实色彩3D全息图.
- 在计算成本,焦点控制和动态全息显示屏的遮蔽方面克服传统CGH方法的局限性.
主要方法:
- 在大型CGH数据集 (MIT-CGH-4K) 上开发并训练了一种记忆效率高的卷积神经网络 (CNN),其中包括4000个RGB深度图像和全息图对.
- 该CNN使用可微分波式损失函数进行训练,近似弗雷内尔衍射,并采用反化仅相位编码方法.
- 该管道被优化为消费者GPU的实时性能,并在移动和边缘AI设备上展示交互式速率.
主要成果:
- 深度学习管道实时合成现实色彩的3D全息图 (在GPU上以1920x1080分辨率60Hz).
- 该CNN模型具有高度的内存效率 (<620KB) 并在移动 (1.1Hz) 和边缘 (2.0Hz) 设备上实现交互式速率.
- 使用开发的方法,实验证明了没有斑点的,自然的,高分辨率的3D全息图.
结论:
- 基于学习的CGH方法显著提升了实时3D显示功能,使动态全息技术变得实用.
- 开发的管道和数据集为下一代虚拟和增强现实系统,移动全息应用和先进的光学技术铺平了道路.
- 这项工作可以在诸如超表面设计,微观操纵,全息显微镜和3D打印等领域得到更广泛的应用.
相关概念视频
01:15Depth Perception and Spatial Vision
1.4K
Depth perception is the ability to perceive objects three-dimensionally. It relies on two types of cues: binocular and monocular. Binocular cues depend on the combination of images from both eyes and how the eyes work together. Since the eyes are in slightly different positions, each eye captures a slightly different image. This disparity between images, known as binocular disparity, helps the brain interpret depth. When the brain compares these images, it determines the distance to an object.
1.4K
01:20Deconvolution
403
Deconvolution, also known as inverse filtering, is the process of extracting the impulse response from known input and output signals. This technique is vital in scenarios where the system's characteristics are unknown, and they must be inferred from the observable signals.
Deconvolution involves several mathematical techniques to derive the impulse response. One common approach is polynomial division. In this method, the input and output sequences are treated as coefficients of...
Deconvolution involves several mathematical techniques to derive the impulse response. One common approach is polynomial division. In this method, the input and output sequences are treated as coefficients of...
403