Beyond Corroboration: Strengthening Model Validation by Looking for Unexpected Patterns

Area of Science:

• Complex Systems
• Computational Modeling
• Scientific Validation

Background:

• Emergent phenomena models explain global patterns from local processes.
• Current model validation primarily relies on reproducing known patterns (corroboration).
• There's a need for methods that test model robustness beyond simple pattern reproduction.

Purpose of the Study:

• To propose and evaluate a robust model validation method.
• To introduce an open-ended evolutionary approach for exploring model behavior.
• To assess the method's effectiveness in discovering diverse emergent patterns and identifying model limitations.

Main Methods:

• Developed the Pattern Space Exploration (PSE) method, an open-ended evolutionary approach.
• Utilized Novelty Search to drive the exploration of diverse model outputs.
• Tested PSE on a collective motion model and a city system dynamics model.
• Compared PSE against traditional a priori sampling experiment designs.

Main Results:

• PSE successfully identified all known types of collective motion.
• PSE significantly outperformed a priori sampling methods in pattern discovery.
• Applied to city dynamics, PSE revealed potential predictive scenarios and model falsifiers for unrealistic simulations.

Conclusions:

• Robust model validation necessitates falsification alongside corroboration.
• The proposed Pattern Space Exploration method enhances model validation by systematically exploring behavioral diversity.
• This approach offers valuable insights for understanding complex systems and improving predictive modeling.