A vision chip with complementary pathways for open-world sensing

Area of Science:

• Computer Vision
• Sensor Technology
• Biomimetic Systems

Background:

• Image sensors struggle with dynamic, unpredictable open-world scenes due to power and bandwidth limitations.
• Existing sensors face trade-offs between speed, resolution, dynamic range, and precision.

Purpose of the Study:

• To introduce a novel complementary sensing paradigm inspired by the human visual system.
• To overcome fundamental limitations in developing vision systems for diverse open-world applications.

Main Methods:

• Developed a vision chip, Tianmouc, with a hybrid pixel array and parallel-and-heterogeneous readout architecture.
• Implemented a primitive-based representation to create cognition-oriented and action-oriented pathways.
• Integrated the Tianmouc chip into an autonomous driving system.

Main Results:

• Achieved high-speed sensing up to 10,000 fps and a dynamic range of 130 dB.
• Demonstrated adaptive bandwidth reduction by 90%.
• Enabled accurate, fast, and robust perception in challenging autonomous driving scenarios.

Conclusions:

• The primitive-based complementary sensing paradigm effectively addresses limitations in current image sensor technology.
• The Tianmouc chip offers superior performance in spatial resolution, speed, and dynamic range for open-world applications.
• This approach enhances perception capabilities for autonomous systems in complex environments.