A Concave Pairwise Fusion Approach to Heterogeneous Q-Learning for Dynamic Treatment Regimes

A dynamic treatment regime is a sequence of decision rules that map available history information to a treatment option at each decision point. The optimal dynamic treatment regime seeks to make these decisions to maximize the expected outcome of interest. Most existing methods assume population homogeneity. In many complex applications, ignoring latent heterogeneous structures may compromise estimation, highlighting the necessity of exploring heterogeneous structures during the estimation of optimal treatment regimes. We propose heterogeneous Q-learning that facilitates the estimation of optimal dynamic treatment regimes using a concave pairwise fusion penalized approach. The proposed method employs an alternating direction method of multipliers algorithm to solve the concave pairwise fusion penalized least squares problem in each stage. Simulation studies demonstrate that our proposed method outperforms the standard Q-learning method, and it is further illustrated through a real data analysis from the China Rural Hypertension Control Project (CRHCP) study group.