WTMAD-4: a fair weighting scheme for GMTKN55

Area of Science:

• Molecular quantum chemistry
• Computational chemistry
• Electronic structure theory

Background:

• The GMTKN55 dataset is a standard benchmark for evaluating electronic-structure methods in molecular quantum chemistry.
• Current performance metrics, like weighted mean absolute deviation (WTMAD), unfairly under-weight certain benchmarks within the GMTKN55 set.
• This can lead to inaccurate assessments of method performance, particularly for density-functional approximations (DFAs).

Purpose of the Study:

• To identify and address the flaw in existing WTMAD definitions used for the GMTKN55 dataset.
• To propose a new, more equitable performance metric (WTMAD-4) that properly accounts for all component benchmarks.
• To reassess the performance of a large number of DFAs using the improved metric.

Main Methods:

• Analysis of existing WTMAD definitions and their impact on GMTKN55 benchmark weighting.
• Development of the WTMAD-4 metric, calibrated using errors from ten minimally empirical dispersion-corrected DFAs.
• Reassessment of 115 DFAs using the WTMAD-4 metric.

Main Results:

• A significant under-weighting of certain GMTKN55 benchmarks by conventional WTMAD definitions was confirmed.
• The proposed WTMAD-4 metric provides a more balanced and fair evaluation across all GMTKN55 benchmarks.
• Reassessment revealed instances where DFAs optimized with WTMAD-2 show suboptimal performance on benchmarks marginalized by that metric.

Conclusions:

• The WTMAD-4 metric offers a more reliable way to assess electronic-structure method performance on the GMTKN55 dataset.
• Fairer evaluation is crucial for developing accurate and robust density-functional approximations.
• Existing optimization strategies may need revision to account for the limitations of older WTMAD metrics.