CANDLE/Supervisor: a workflow framework for machine learning applied to cancer research.

Area of Science:

• Computational science
• Machine learning
• Bioinformatics

Background:

• Supercomputers present challenges for large machine learning workflows due to complex algorithms and data flows.
• Managing large-scale experiments on high-performance computing systems is difficult.

Purpose of the Study:

• To present a workflow system, CANDLE/Supervisor, for scaling machine learning ensembles.
• To address challenges in hyper-parameter exploration for deep neural networks.

Main Methods:

• Development of the CANDLE/Supervisor workflow system.
• Application of the system to deep neural network ensembles.

Main Results:

• The system demonstrates progress in scaling machine learning ensembles, specifically deep neural networks.
• CANDLE/Supervisor addresses hyper-parameter exploration for deep neural networks.

Conclusions:

• Initial results show successful scaling and multi-platform execution of CANDLE on DOE systems (ORNL, ANL, NERSC).
• The system is effective on large-scale scientific computing resources.