Offline reinforcement learning for learning to dispatch for job shop scheduling.

Area of Science:

• Operations Research
• Artificial Intelligence
• Machine Learning

Background:

• The Job Shop Scheduling Problem (JSSP) is a significant combinatorial optimization challenge.
• Online Reinforcement Learning (RL) offers rapid solutions for JSSP but suffers from sample inefficiency and requires impractical simulated environments.
• Existing methods like Constraint Programming (CP) provide high-quality solutions but are not easily integrated with RL.

Purpose of the Study:

• To introduce an offline reinforcement learning approach, Offline Learned Dispatching (Offline-LD), for the Job Shop Scheduling Problem (JSSP).
• To address the limitations of online RL, including sample inefficiency and the need for simulated environments, by leveraging historical scheduling data.
• To enable the utilization of pre-existing high-quality solutions from methods like CP within an RL framework.

Main Methods:

• Developed maskable variants of Q-learning methods: Maskable Quantile Regression DQN (mQRDQN) and discrete maskable Soft Actor-Critic (d-mSAC).
• Employed Conservative Q-Learning (CQL) to enable learning from historical data.
• Introduced a novel entropy bonus modification for d-mSAC in maskable action spaces and a new reward normalization technique for offline RL in JSSP.

Main Results:

• Offline-LD demonstrated superior performance compared to online RL on both generated and benchmark JSSP instances.
• The approach achieved strong results even when trained on a limited set of 100 solutions generated by CP.
• Introducing noise into the expert dataset resulted in performance comparable or superior to using the clean expert dataset, indicating robustness.

Conclusions:

• Offline-LD effectively addresses key limitations of online RL for JSSP by learning from historical data.
• The method shows promise for real-world applications where historical data is available and potentially noisy.
• Offline-LD offers a viable alternative for complex scheduling environments where online training is impractical.