Set Norm and Equivariant Skip Connections: Putting the Deep in Deep Sets

Area of Science:

• Machine Learning
• Deep Learning
• Set Theory

Background:

• Permutation invariant neural networks are effective for set predictions.
• Existing architectures like Deep Sets and Set Transformer face challenges with deep networks, including vanishing/exploding gradients.
• Layer normalization can hinder performance in Set Transformer by removing predictive information.

Purpose of the Study:

• To address gradient instability and performance degradation in deep permutation invariant neural networks.
• To introduce novel architectural improvements for handling set data.
• To develop and validate enhanced models for set prediction tasks.

Main Methods:

• Introduced the "clean path principle" for equivariant residual connections.
• Developed "set norm" (sn), a novel normalization technique specifically for sets.
• Built and evaluated Deep Sets++ and Set Transformer++ models.

Main Results:

• The new models, Deep Sets++ and Set Transformer++, achieve high depths.
• Performance is comparable or superior to original architectures across various tasks.
• Introduced Flow-RBC, a new single-cell dataset for permutation invariant prediction.

Conclusions:

• The "clean path principle" and set norm effectively enable deeper and more performant permutation invariant networks.
• Deep Sets++ and Set Transformer++ represent significant advancements in set prediction.
• The new dataset facilitates real-world applications in areas like single-cell analysis.