Fully neuromorphic vision and control for autonomous drone flight.

Area of Science:

• Robotics
• Neuromorphic Engineering
• Artificial Intelligence

Background:

• Biological systems achieve low-latency, energy-efficient perception and action through asynchronous, sparse processing.
• Neuromorphic hardware and spiking neural networks (SNNs) aim to replicate these characteristics in robotics.
• Current robotic applications of SNNs are limited by processor constraints and training complexities.

Purpose of the Study:

• To present a fully neuromorphic vision-to-control pipeline for autonomous drone flight.
• To demonstrate the capability of SNNs to process raw event-based camera data for real-time control.
• To achieve efficient, low-power operation for onboard robotic systems.

Main Methods:

• A five-layer SNN with 28,800 neurons was trained using self-supervised learning to map event data to ego-motion estimates.
• A single decoding layer for control actions was trained using an evolutionary algorithm in a drone simulator.
• The pipeline was implemented and tested on Intel's Loihi neuromorphic processor for sim-to-real transfer.

Main Results:

• The neuromorphic pipeline successfully enabled autonomous vision-based flight, including hovering, landing, and complex maneuvers like simultaneous sideways movement and yawing.
• The system demonstrated accurate ego-motion control in robotic experiments, validating the sim-to-real transfer.
• The onboard implementation achieved high efficiency, running at 200 Hz with minimal power consumption (0.94W idle, 7-12mW additional when active).

Conclusions:

• A fully neuromorphic, event-driven vision-to-control system can achieve complex autonomous flight in drones.
• This approach offers a pathway towards highly energy-efficient and low-latency robotic perception and control.
• The results highlight the potential of neuromorphic processing for creating insect-sized intelligent robots.