A Multi-Modal, Multi-Temporal, Multi-Resolution Benchmark Dataset for Building Height Estimation

Building heights are crucial for sustainable urban planning and monitoring. While traditional methods use airborne stereo images and LiDAR data for accurate height estimation, their large-scale application is costly and slow, limiting the ability to conduct frequent large-scale monitoring. In contrast, satellite data offers a scalable alternative, further improved with Deep Learning (DL) models. However, the lack of representative open-source training datasets has constrained the progress in this field. In this paper, we introduce M4Heights, a multi-modal, multi-resolution, and multi-temporal dataset designed for building height estimation, spanning diverse architectural styles, urban densities, and terrain complexities across Estonia, Netherlands, and Switzerland. The dataset includes  ≈ 1 million images, comprising time series of Sentinel-1 SAR and Sentinel-2 MSI satellite data, high-resolution aerial orthophotos, and high-quality building height reference maps. Additionally, M4Heights provides the largest associated multi-image super-resolution dataset to enhance height estimation accuracy. Our dataset supports a range of modeling approaches, offers extensibility to new geographic regions and provides opportunities to advance the development of DL models for building height estimation.