Group Maximum Differentiation Competition: Model Comparison with Few Samples.

Area of Science:

• Computational Science and Engineering
• Data Science
• Perceptual Science

Background:

• Selecting optimal computational models is critical in science and engineering, especially when physical measurements are scarce.
• High-dimensional data spaces, like those in visual quality prediction, present significant challenges for model comparison.
• Existing methods may struggle with limited sample sets and vast content variations.

Purpose of the Study:

• To introduce a novel methodology, group maximum differentiation (gMAD) competition, for robust computational model comparison.
• To address the challenge of selecting the best predictive model under constraints of limited physical measurements.
• To provide a framework for evaluating model performance in terms of both offensive (attacking) and defensive capabilities.

Main Methods:

• The gMAD competition framework treats multiple computational models as both 'attackers' and 'defenders'.
• It identifies sample pairs that maximally differentiate 'attacker' models while keeping a 'defender' model constant.
• Introduces 'aggressiveness' and 'resistance' metrics to quantify model performance in the competition.

Main Results:

• Demonstrated the gMAD competition's effectiveness using examples from image quality, image aesthetics, and video quality-of-experience.
• The methodology successfully identified differentiating sample pairs within high-dimensional visual data.
• Quantified model performance through aggressiveness and resistance metrics.

Conclusions:

• gMAD competition offers a powerful approach for model selection in data-limited scenarios.
• The methodology is particularly valuable when physical measurements are expensive and computational prediction is low-cost.
• gMAD is extendable to diverse scientific and engineering fields beyond visual perception.