The Challenges and Limitations of Validating Satellite-Derived Datasets Using Independent Measurements: Lessons Learned from Essential Climate Variables

Validation of satellite-derived essential climate variable (ECV) datasets requires comparison against independent measurements. These independent measurements, which include ground-based, airborne, and other non-satellite-based measurements, are typically the product of a different measurement system and may include some contribution from models. These reference data therefore have their own characteristics, uncertainties, and limitations which must be accounted for in the validation process. In addition, they typically differ from the data to be validated in spatio-temporal resolution, sensitivity, and sampling. As such, comparisons to independent data do not necessarily yield clear feedback on the quality of satellite data and insufficient awareness of these issues can lead to erroneous interpretation. This is the cost of leaving the laboratory and studying the real world. In this review paper, we examine the challenges and limitations of evaluating satellite-derived datasets with independent measurements, using examples across different ECVs within the terrestrial, ocean, and atmospheric domains. Drawing from other studies, we discuss issues with the reference datasets themselves, issues specific to use of these data for validation, and issues resulting from the comparison methodology. We conclude with recommendations to the community based on this review. In this, we highlight the importance of continued efforts towards (1) advancing uncertainty modelling of reference datasets and quality control knowledge and procedures, (2) establishing and communicating limitations in reference data, (3) reference data (and metadata) timeliness and preservation, and (4) best practices for the validation methodologies that address the spatio-temporal differences of the measurements.