Exploring the Utility of Developer Exhaust.

Area of Science:

• Computer Science
• Machine Learning
• Artificial Intelligence

Background:

• Machine learning data analysis generates "developer exhaust" (code, logs, metadata) as byproducts.
• This exhaust contains rich information that can be leveraged for complex tasks.

Purpose of the Study:

• To investigate using developer exhaust, specifically log data from deep learning model training, for efficient model search.
• To predict performance metrics of untrained models using only log data.

Main Methods:

• Developed two preliminary methods: a nearest neighbor approach with edit distance for comparing architectures and an LSTM-based end-to-end approach.
• Trained an LSTM using model architectures and logs to predict performance metrics like validation accuracy and training time.

Main Results:

• Achieved an average prediction error of 1.37% for validation accuracy, significantly outperforming a baseline of 4.13%.
• Selected top-3 models within training time constraints with 82% overlap with true top models, compared to 54% for the baseline.

Conclusions:

• Developer exhaust provides valuable data for learning models that can efficiently approximate complex tasks.
• The proposed methods demonstrate the potential of log data for accurate and efficient model search in deep learning.