IBCB: Efficient Inverse Batched Contextual Bandit for Behavioral Evolution History.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Reinforcement Learning

Background:

• Traditional imitation learning requires fixed expert data, insufficient for streaming applications with evolving decision-makers.
• Online learning in streaming systems generates interaction histories reflecting novice-to-expert behavioral changes.
• Existing imitation learning methods struggle with data from evolving experts.

Purpose of the Study:

• Propose an inverse batched contextual bandit (IBCB) framework for imitation learning.
• Address the challenge of learning from experts with evolving behavior in streaming applications.
• Enable efficient estimation of reward parameters and policies from behavioral evolution history.

Main Methods:

• Formulated the inverse problem as a quadratic programming problem using behavioral evolution history.
• Developed the inverse batched contextual bandit (IBCB) framework for batched contextual bandits with inaccessible rewards.
• Extended IBCB for fairness-aware expert imitation.

Main Results:

• IBCB demonstrated superior performance compared to existing imitation learning algorithms on synthetic and real-world datasets.
• IBCB significantly reduced running time and showed improved imitation for fairness-aware experts.
• Empirical analyses confirmed IBCB's effectiveness in learning from novice expert interaction history and its out-of-distribution generalization capabilities.

Conclusions:

• IBCB is a unified framework for deterministic and randomized bandit policies, adept at handling expert behavioral evolution.
• The proposed framework offers a significant advancement in imitation learning for streaming applications.
• IBCB provides a robust and efficient solution for learning from evolving expert data, with implications for fairness and generalization.