autonlab/weasel

Weakly Supervised End-to-End Learning (NeurIPS 2021)

WeaSEL: Weakly Supervised End-to-end Learning

This is a PyTorch-Lightning-based framework, based on our End-to-End Weak Supervision paper (NeurIPS 2021), that allows you to train your favorite neural network for weakly-supervised classification¹

• only with multiple labeling functions (LFs)², i.e. without any labeled training data!
• in an end-to-end manner, i.e. directly train and evaluate your neural net (end-model from here on), there's no need to train a separate label model any more as in Snorkel & co,
• with better test set performance and enhanced robustness against correlated or inaccurate LFs than prior methods like Snorkel

¹ This includes learning from crowdsourced labels or annotations!
² LFs are labeling heuristics, that output noisy labels for (subsets of) the training data (e.g. crowdworkers or keyword detectors).

If you use this code, please consider citing our work

End-to-End Weak Supervision
Salva Rühling Cachay, Benedikt Boecking, and Artur Dubrawski
Advances in Neural Information Processing Systems (NeurIPS), 2021
arXiv:2107.02233v3

Credits

• The following template was extremely useful as source of inspiration and for getting started with the PL+Hydra implementation: ashleve/lightning-hydra-template

• Weasel image credits go to Rohan Chang for this Unsplash-licensed image

Getting Started

This library assumes familiarity with (multi-source) weak supervision, if that's not the case you may want to first learn its basics in e.g. this overview slides from Stanford or this Snorkel tutorial.

That being said, have a look at our examples and the notebooks therein showing you how to use Weasel for your own dataset, LF set, or end-model. E.g.:

• A high-level starter tutorial, with few code, many explanations and including Snorkel as a baseline (so that if you are familiar with Snorkel you can see the similarities and differences to Weasel).

• See how the whole WeaSEL pipeline works with all details, necessary steps and definitions for a new dataset & custom end-model. This notebook will probably make you learn the most about WeaSEL and how to apply it to your own problem.

• A realistic ML experiment script with all that's part of a ML pipeline, including logging to Weight&Biases, arbitrary callbacks, and eventually retrieving your fully trained end-model.

Reproducibility

Please have a look at the research code branch, which operates on pure PyTorch.

Installation

1. New environment (recommended, but optional)

conda create --name weasel python=3.9
conda activate weasel  

2a: From source

python -m pip install git+https://github.com/autonlab/weasel#egg=weasel[all]

2b: From source, editable install

git clone https://github.com/autonlab/weasel.git
cd weasel
pip install -e .[all]

Minimal dependencies

Minimal dependencies, in particular not using Hydra, can be installed with

python -m pip install git+https://github.com/autonlab/weasel

The needed environment corresponds to conda env create -f env_gpu_minimal.yml.

If you choose to use this variant, you won't be able to run some of the examples: You may want to have a look at this notebook that walks you through how to use Weasel without Hydra as the config manager.

Note: Weasel is under active development, some uncovered edge cases might exist, and any feedback is very welcomed!

Apply WeaSEL to your own problem

Configuration with Hydra

Optional: This template config will help you get started with your own application, an analogous config is used in this tutorial script that you may want to check out.

Pre-defined or custom downstream models & Baselines

Please have a look at the detailed instructions in this Readme.

Using your own dataset and/or labeling heuristics

Please have a look at the detailed instructions in this Readme.

Citation

@article{cachay2021endtoend,
  author={R{\"u}hling Cachay, Salva and Boecking, Benedikt and Dubrawski, Artur},
  journal={Advances in Neural Information Processing Systems}, 
  title={End-to-End Weak Supervision},
  year={2021}
}

weasel's Issues

About the "strange error" in tutorial notebook

Thanks for the excellent work and code! When I try to run the tutorial 1_bias_bios, I face the same error mentioned in the notebook (ValueError: dictionary update sequence element #0 has length 1; 2 is required). Although rerunning the cell solves the problem, such a solution is not available when running the python code directly. Do you have any hints on what might cause the problem, and what can be done to fix it? It seems to me that the error might be related to hydra, since the corresponding part 1_bias_bios_no_hydra works fine for me.

Broken links to examples files

Thanks for the work! I would like to point out that the links to the tutorials in this page are broken, location of files is not correct, just in case you are not aware of it.

Multi-Label

Is there anything to consider for multi-labelling problems?

Errors in example scripts and notebook

Hello, thank you for sharing this work! I've been trying to run the example scripts and notebooks before running weasel on a different dataset. I keep on coming across the following exceptions when trying to fit weasel and the end-model:

1. TypeError in 0_full_pipeline.ipynb

TypeError: vars() argument must have dict attribute

2. MisconfigurationException in all other files in the examples folder

lightning_fabric.utilities.exceptions.MisconfigurationException: You are trying to self.log() but it is not managed by the Trainer control flow

Do you know what might be the problem?

raw data?

Hi,

Great work! Just wondering if the raw data, eg, raw text for IMDb and Amazon could be also released?

Request for Labeling Functions

Thanks for open-sourcing the code! Quick question: where can I find the additional labeling functions for Amazon, IMDB (136 LFs), and BiasBios (99 LFs)?