Adaptive exploration strategy in reinforcement learning based on Q-values and environmental cognition

Reinforcement learning has achieved strong results across diverse sequential decision making problems, but balancing exploration and exploitation to avoid local optima remains a core challenge. Many methods rely on a single signal, such as visitation counts, which yields indiscriminate exploration and fails to focus on key decisions with large value differences. We propose Var, an adaptive exploration strategy composed of two parts: Q-values and environmental cognition. Environmental cognition consists of two signals, a Q-value difference term and a novelty bonus. Using these two signals, Var encourages broad and deep exploration early in training and then transitions to an exploitation phase driven by Q-values. To apply the strategy across environments with different state-space characteristics, we integrate it into classical algorithms and introduce Var-QL and Var-DQN. For large tabular discrete environments, we propose LRU-Var-QL, which augments Var-QL with a Least Recently Used (LRU) cache to avoid catastrophic actions. On FrozenLake-v1, LRU-Var-QL reduces catastrophic actions; across nine Atari games, the proposed methods achieve higher cumulative returns and faster learning than strong exploration baselines.