Deep Learning Improves Parameter Estimation in Reinforcement Learning Models.

Area of Science:

• Cognitive science
• Computational neuroscience
• Machine learning

Background:

• Cognitive models are essential tools in psychology and neuroscience for understanding cognitive processes.
• Reliable parameter estimation is crucial for scientific inference from these models, but often challenging, especially with limited data.
• Parameter ambiguity arises when multiple parameter sets fit data equally well, questioning the scientific meaningfulness of estimates.

Purpose of the Study:

• To investigate parameter ambiguity in reinforcement learning models using two distinct optimization methods.
• To compare the performance of a traditional optimization method (Nelder-Mead) with a deep learning pipeline for parameter estimation.
• To introduce and apply a systematic evaluation framework beyond predictive accuracy to assess parameter reliability.

Main Methods:

• Applied the Nelder-Mead (fminsearch) optimization method and a deep learning pipeline to estimate parameters across ten decision-making datasets.
• Developed a novel evaluation framework assessing generalizability, robustness, identifiability, and test-retest reliability.
• Compared parameter estimates from both methods using the proposed evaluation framework.

Main Results:

• Both optimization methods achieved similar fitting performance but yielded substantially different parameter estimates.
• The deep learning pipeline demonstrated superior performance across metrics of generalizability, robustness, identifiability, and test-retest reliability compared to Nelder-Mead.
• Parameter ambiguity was consistently observed across datasets, highlighting its significance.

Conclusions:

• Parameter ambiguity is a critical, underappreciated challenge impacting scientific replicability in cognitive modeling.
• The choice of optimization method significantly influences scientific conclusions derived from cognitive models.
• A multi-faceted evaluation approach and integration of deep learning pipelines are recommended for reliable scientific inference.