Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale

Area of Science:

• Machine Learning
• Data Science
• Software Engineering

Background:

• Labeling training data is a major bottleneck in developing machine learning applications, significantly increasing development time and cost.
• Existing organizational knowledge resources are often underutilized in the machine learning development lifecycle.

Purpose of the Study:

• To investigate the use of organizational knowledge as weak supervision to reduce machine learning development costs and time.
• To introduce Snorkel DryBell, a novel weak supervision management system tailored for organizational knowledge integration.
• To evaluate the effectiveness of Snorkel DryBell on real-world classification tasks.

Main Methods:

• Leveraging existing organizational knowledge resources as weak supervision signals.
• Developing Snorkel DryBell, a system featuring template-based knowledge ingestion, cross-feature production serving, and scalable, sampling-free execution.
• Applying Snorkel DryBell to three classification tasks within Google.

Main Results:

• Snorkel DryBell achieved comparable classifier quality to models trained with tens of thousands of hand-labeled examples.
• The system converted non-servable organizational resources into servable models, yielding an average 52% performance improvement.
• Execution over millions of data points was achieved in tens of minutes, demonstrating scalability.

Conclusions:

• Organizational knowledge can be effectively utilized as weak supervision to drastically reduce machine learning development costs and time.
• Snorkel DryBell provides a scalable and efficient solution for managing weak supervision from diverse organizational knowledge sources.
• The developed system demonstrates significant performance improvements and efficiency gains in practical machine learning applications.