Snuba: Automating Weak Supervision to Label Training Data

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Deep learning models require extensive high-quality training data, a significant bottleneck for diverse applications.
• Weak supervision, using imperfect label sources like heuristics, is a common alternative but requires manual design for each task.
• Manual heuristic design is time-consuming and resource-intensive, involving domain experts in repetitive tasks.

Purpose of the Study:

• To present Snuba, a system for automatically generating heuristics in the weak supervision setting.
• To address the challenges of manual heuristic design by automating the creation of labeling functions.
• To improve the efficiency and effectiveness of generating training labels for large unlabeled datasets.

Main Methods:

• Snuba utilizes a small labeled dataset to generate heuristics that label subsets of a large unlabeled dataset.
• It iteratively generates and refines heuristics until a substantial portion of the data is labeled.
• A statistical measure ensures the termination of the iterative process, maintaining label quality.

Main Results:

• Snuba automatically generates heuristics in under five minutes.
• The system achieved up to 9.74 F1 points improvement over the best user-defined heuristics.
• Snuba outperformed semi-supervised learning approaches by up to 14.35 F1 points in real-world collaborations.

Conclusions:

• Snuba offers an automated and efficient solution for generating high-quality weak supervision labels.
• The system significantly reduces the time and cost associated with preparing training data for deep learning.
• Snuba demonstrates superior performance compared to both manual heuristic design and other automated methods.