Dynamic sparse attention for lightweight multimodal sensor fusion on edge devices.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Edge Computing

Background:

• Multimodal sensor fusion for edge computing demands efficient attention mechanisms.
• Standard dense attention models face computational and memory limitations on resource-constrained devices.

Purpose of the Study:

• To introduce a Dynamic Sparse Attention (DSA) module and deployment pipeline for efficient edge-side multimodal fusion.
• To enable real-time attention processing within tight hardware budgets.

Main Methods:

• Developed an entropy-conditioned predictor for input-adaptive attention masks, focusing computation on relevant cross-modal pairs.
• Implemented a heterogeneous feature alignment block with modality-adaptive normalization to reconcile diverse sensor data.
• Combined block-sparse reformulation, mixed-precision quantization, and hardware-aware scheduling for optimized deployment.

Main Results:

• DSA achieved performance within 0.9% mAP of dense Transformer baselines on nuScenes and KITTI datasets.
• Inference latency was reduced by over 50%, and peak memory usage by approximately 45% on an NVIDIA Jetson Orin NX.
• Ablation studies validated the contribution of each component, demonstrating graceful degradation with sensor dropout.

Conclusions:

• The proposed DSA module and deployment pipeline offer a computationally efficient solution for multimodal sensor fusion in edge computing.
• DSA enables significant reductions in latency and memory footprint without substantial performance compromise.
• The approach is robust and adaptable to dynamic sensor availability in real-world edge applications.