g-Distance: On the comparison of model and human heterogeneity

Area of Science:

• Cognitive Science
• Computational Modeling
• Psychology

Background:

• Traditional model evaluation often assumes uniform human and model behavior.
• This overlooks the inherent heterogeneity present in both human and model responses.

Purpose of the Study:

• Introduce a novel metric, g-distance, for evaluating computational model adequacy.
• Assess model performance by considering the range of behaviors exhibited by both models and humans.

Main Methods:

• Define g-distance based on accommodation and excess flexibility.
• Apply g-distance to five models of the inverse base-rate effect.
• Compare g-distance results with the Bayesian information criterion.

Main Results:

• g-distance identifies two models, NNRAS and DGCM18, as superior to the EXIT model.
• This finding remains consistent across various weightings of accommodation and excess flexibility.
• The Bayesian information criterion incorrectly identifies a poor-performing model as the most adequate.

Conclusions:

• g-distance offers a more nuanced evaluation of computational models than traditional metrics.
• Formalizing theories is crucial, as model behavior can be counterintuitive.
• This work highlights the need for improved methods in computational modeling and model evaluation.