β-free Information Bottleneck in Representation Learning

The bottleneck size β is a crucial parameter that governs the trade-off between information compression and information preservation in information bottleneck (IB) theory. Existing technologies fail to automatically learn an optimal β and instead rely on a "pre-specified β" paradigm. This paradigm compromises the generalization capability of the IB principle, making it effective only in specific task scenario and severely hindering the real-world applicability. In addition, the "pre-specified β" paradigm treats β as an isolated hyperparameter, which prevents synergy with multiple contextual hyperparameters and thereby impairs the balance of information compression and preservation. In this paper, we take the first step toward addressing this unprecedented challenge by proposing a novel β-free IB method with a profound and meaningful divide-and-conquer strategy. Specifically, we formulate β as comprising the static bottleneck size and the super bottleneck size inspired from dam crest elevation, where the former captures the intrinsic relation between β and static factors (e.g., task labels) and the latter discovers the dynamic synergy between β and dynamic factors (e.g., hyperparameters). In this way, an optimal β is obtained to guide the IB method in learning compact representation achieving an ideal balance. As a key theoretical contribution of this work, we systematically derive that the β-free IB can obtain compact representation guided by the heuristic β from the perspective of phase transitions. To evaluate the effectiveness of the proposed method, we conduct extensive image recognition and text classification experiments on benchmark datasets. Our β-free IB offers a more effective strategy to learn compact representation, and has the potential to stimulate a novel research on IB-based pattern recognition. The code is available at https://github.com/SWT-AITeam/beta-free.