Optimal dose escalation methods using deep reinforcement learning in phase I oncology trials

In phase I trials of a novel anticancer drug, one of the most important objectives is to identify the maximum tolerated dose (MTD). To this end, a number of methods have been proposed and evaluated under various scenarios. However, the percentages of correct selection (PCS) of MTDs using previous methods are insufficient to determine the dose for late-phase trials. The purpose of this study is to construct an action rule for escalating or de-escalating the dose and continuing or stopping the trial to increase the PCS as much as possible. We show that deep reinforcement learning with an appropriately defined state, action, and reward can be used to construct such an action selection rule. The simulation study shows that the proposed method can improve the PCS compared with the 3 + 3 design, CRM, BLRM, BOIN, mTPI, and i3 + 3 methods.