TAMC:为开放的词汇3D场景理解提供文本对齐和掩盖一致性
Juan Wang1, Zhijie Wang2, Tomo Miyazaki1
1Department of Communications Engineering, Graduate School of Engineering, Tohoku University, Sendai 9808579, Japan.
Sensors (Basel, Switzerland)
|October 16, 2024
概括
本研究引入了使用点云数据进行3D场景理解的新方法. 通过使用掩盖一致性培训和伪文本监督,该方法增强了对虚拟现实和机器人等应用程序的环境感知.
科学领域:
- 计算机视觉 计算机视觉
- 机器人技术 机器人技术 机器人技术
- 虚拟现实 虚拟现实 虚拟现实
背景情况:
- 三维 (3D) 场景理解依赖于分析环境感知点云数据.
- 现有的方法将二维图像特征与三维点云特征对齐,以获得开放的词汇理解.
- 目前的方法面临着稀疏/不完整3D数据的挑战,在培训期间缺乏直接文本监督.
研究的目的:
- 为了解决从稀疏/不完整的点云中提取3D特征的缺陷.
- 通过直接的文本监督,提高培训和推断阶段之间的一致性.
- 为了增强开放的词汇3D场景理解能力.
主要方法:
- 实施了掩盖一致性培训政策,通过掩盖部分数据来处理稀疏的3D特征.
- 通过从融合的2D图像描述创建场景描述来生成伪文本标签.
- 在培训过程中同时对准2D-3D功能和3D文本功能.
主要成果:
- 提出的方法有效地解决了稀疏和不完整的3D点云数据的挑战.
- 在培训期间直接监督文本可以提高模型的理解和一致性.
- 实验结果表明,与最先进的方法相比,性能优越.
结论:
- 这种新的方法通过克服现有方法的局限性,大大提高了3D场景的理解.
- 掩盖一致性和伪文监控的结合为现实世界的场景提供了强大的解决方案.
- 这项工作为虚拟现实,机器人及其它领域的环境感知提供了更有效的方法.
相关概念视频
Masking and Demasking Agents
2.4K
EDTA titrations may necessitate masking and demasking agents to temporarily protect a particular metal ion in a mixture from the EDTA reaction. These agents facilitate the sequential analysis of the metal ions by forming stable complexes with some—but not all—metal ions during certain steps.
There are many masking agents, such as cyanide, fluoride, triethanolamine, thiourea, and 2,3-bis(sulfanyl)propan-1-ol (formerly 2,3-dimercapto-1-propanol), with the masking agent chosen based on...
There are many masking agents, such as cyanide, fluoride, triethanolamine, thiourea, and 2,3-bis(sulfanyl)propan-1-ol (formerly 2,3-dimercapto-1-propanol), with the masking agent chosen based on...
2.4K
Modeling and Similitude
247
Scaled modeling is a fundamental technique in engineering, enabling the study of large and complex systems by creating smaller, manageable replicas that recreate critical characteristics of the original. In hydrology and civil infrastructure, for example, scaled models of dams help analyze water flow, turbulence, and pressure. This method allows for accurate predictions of real-world behavior within a controlled environment, significantly reducing the cost and time involved in full-scale...
247
Depth Perception and Spatial Vision
601
Depth perception is the ability to perceive objects three-dimensionally. It relies on two types of cues: binocular and monocular. Binocular cues depend on the combination of images from both eyes and how the eyes work together. Since the eyes are in slightly different positions, each eye captures a slightly different image. This disparity between images, known as binocular disparity, helps the brain interpret depth. When the brain compares these images, it determines the distance to an object.
601
Improving Translational Accuracy
9.3K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
9.3K
Three-Compartment Open Model
156
The three-compartment open model is a pharmacokinetic model used to describe the distribution and elimination of drugs following extravascular administration. It comprises a central compartment representing the plasma and two peripheral compartments. The highly perfused peripheral compartment represents organs and tissues with a rich blood supply, such as the liver, kidneys, and lungs. The scarcely perfused peripheral compartment represents tissues with lower blood supply, such as adipose...
156
Relative Motion Analysis using Rotating Axes-Problem Solving
390
Consider a crane whose telescopic boom rotates with an angular velocity of 0.04 rad/s and angular acceleration of 0.02 rad/s2. Along with the rotation, the boom also extends linearly with a uniform speed of 5 m/s. The extension of the boom is measured at point D, which is measured with respect to the fixed point C on the other end of the boom. For the given instant, the distance between points C and D is 60 meters.
Here, in order to determine the magnitude of velocity and acceleration for point...
Here, in order to determine the magnitude of velocity and acceleration for point...
390


