dcai-course/dcai-lab

Lab assignments for Introduction to Data-Centric AI, MIT IAP 2024 👩🏽‍💻

Lab assignments for Introduction to Data-Centric AI

This repository contains the lab assignments for the Introduction to Data-Centric AI class.

Contributions are most welcome! If you have ideas for improving the labs, please open an issue or submit a pull request.

If you're looking for the 2023 version of the labs, check out the 2023 branch.

Lab 1: Data-Centric AI vs. Model-Centric AI

The first lab assignment walks you through an ML task of building a text classifier, and illustrates the power (and often simplicity) of data-centric approaches.

Lab 2: Label Errors

This lab guides you through writing your own implementation of automatic label error identification using Confident Learning, the technique taught in today’s lecture.

Lab 3: Dataset Creation and Curation

This lab assignment is to analyze an already collected dataset labeled by multiple annotators.

Lab 4: Data-centric Evaluation of ML Models

This lab assignment is to try improving the performance of a given model solely by improving its training data via some of the various strategies covered here.

Lab 5: Class Imbalance, Outliers, and Distribution Shift

The lab assignment for this lecture is to implement and compare different methods for identifying outliers. For this lab, we've focused on anomaly detection. You are given a clean training dataset consisting of many pictures of dogs, and an evaluation dataset that contains outliers (non-dogs). Your task is to implement and compare various methods for detecting these outliers. You may implement some of the ideas presented in today's lecture, or you can look up other outlier detection algorithms in the linked references or online.

Lab 6: Growing or Compressing Datasets

This lab guides you through an implementation of active learning.

Lab 7: Interpretability in Data-Centric ML

This lab guides you through finding issues in a dataset’s features by applying interpretability techniques.

Lab 8: Encoding Human Priors: Data Augmentation and Prompt Engineering

[This lab] guides you through prompt engineering, crafting inputs for large language models (LLMs). With these large pre-trained models, even small amounts of data can make them very useful. This lab is also available on Colab.

Lab 9: Data Privacy and Security

The lab assignment for this lecture is to implement a membership inference attack. You are given a trained machine learning model, available as a black-box prediction function. Your task is to devise a method to determine whether or not a given data point was in the training set of this model. You may implement some of the ideas presented in today’s lecture, or you can look up other membership inference attack algorithms.

License

Copyright (c) by the instructors of Introduction to Data-Centric AI (dcai.csail.mit.edu).

dcai-lab is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

dcai-lab is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

See GNU Affero General Public LICENSE for details.

dcai-lab's Issues

Error when running solution to lab 1

I ran into an error while running the solution for lab 1,

File ~/envs/nlpkernel/lib/python3.10/site-packages/transformers/training_args.py:1340, in TrainingArguments.__post_init__(self)
   1334     if version.parse(version.parse(torch.__version__).base_version) == version.parse("2.0.0") and self.fp16:
   1335         raise ValueError("--optim adamw_torch_fused with --fp16 requires PyTorch>2.0")
   1337 if (
   1338     self.framework == "pt"
   1339     and is_torch_available()
-> 1340     and (self.device.type != "cuda")
   1341     and (get_xla_device_type(self.device) != "GPU")
   1342     and (self.fp16 or self.fp16_full_eval)
   1343 ):
   1344     raise ValueError(
   1345         "FP16 Mixed precision training with AMP or APEX (`--fp16`) and FP16 half precision evaluation"
   1346         " (`--fp16_full_eval`) can only be used on CUDA devices."
   1347     )
   1349 if (
   1350     self.framework == "pt"
   1351     and is_torch_available()
   (...)
   1356     and (self.bf16 or self.bf16_full_eval)
   1357 ):

File ~/envs/nlpkernel/lib/python3.10/site-packages/transformers/training_args.py:1764, in TrainingArguments.device(self)
   1760 """
   1761 The device used by this process.
   1762 """
   1763 requires_backends(self, ["torch"])
-> 1764 return self._setup_devices

File ~/envs/nlpkernel/lib/python3.10/site-packages/transformers/utils/generic.py:54, in cached_property.__get__(self, obj, objtype)
     52 cached = getattr(obj, attr, None)
     53 if cached is None:
---> 54     cached = self.fget(obj)
     55     setattr(obj, attr, cached)
     56 return cached

File ~/envs/nlpkernel/lib/python3.10/site-packages/transformers/training_args.py:1672, in TrainingArguments._setup_devices(self)
   1670 if not is_sagemaker_mp_enabled():
   1671     if not is_accelerate_available(min_version="0.20.1"):
-> 1672         raise ImportError(
   1673             "Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1`: Please run `pip install transformers[torch]` or `pip install accelerate -U`"
   1674         )
   1675     AcceleratorState._reset_state(reset_partial_state=True)
   1676 self.distributed_state = None

ImportError: Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1`: Please run `pip install transformers[torch]` or `pip install accelerate -U`

The error message suggested changing this line

!pip install torch transformers datasets

Into

!pip install torch transformers[torch] datasets

or

!pip install torch transformer accelerate datasets

Inconsistency in Dataset Used for acc_original and acc_clean Evaluations in Notebook Lab - Label Errors.ipynb

Hello,

I was reviewing the notebook "Lab - Label Errors.ipynb" and noticed an inconsistency in the datasets used for evaluating acc_original and acc_clean.

In the notebook, the line data = df.drop(issue_idx) results in the removal of rows with index issue_idx from the dataset. This effectively reduces the size of the dataset used for subsequent steps. As a result, the acc_original is evaluated on the full dataset, whereas acc_clean is evaluated on a subset of the original dataset (excluding the rows with index issue_idx).

From my understanding, for a fair comparison between acc_original and acc_clean, we should train without the problematic issue_idx rows but still make predictions on the entire dataset. By excluding these rows for the acc_clean evaluation, it seems like we are leaving out some of the trickiest rows to predict, which might lead to misleading results.

To address this, I've come up with a fix proposal:

# ADDED START
class CL_XGBClassifier(XGBClassifier):
    def __init__(self, idx, **kwargs):
        super().__init__(**kwargs)
        self.idx = idx
        
    def fit(self, X, y):
        X['target'] = y
        for idx in self.idx:
            if idx in X.index:
                X = X.drop(idx)
        y = X.pop('target')
        return super().fit(X, y)
# ADDED END

# Remove the label errors found by Confident Learning
data = df.copy()  # <- MODIFIED
clean_labels = data['noisy_letter_grade']
data = data.drop(['stud_ID', 'letter_grade', 'noisy_letter_grade'], axis=1)

# Train a more robust classifier with less erroneous data
model = CL_XGBClassifier(tree_method="hist", enable_categorical=True, idx=issue_idx)    # <- MODIFIED
clean_pred_probs = cross_val_predict(model, data, clean_labels, method='predict_proba')
clean_preds = np.argmax(clean_pred_probs, axis=1)

acc_clean = accuracy_score(clean_preds, clean_labels)
print(f"Accuracy with original data: {round(acc_original*100, 1)}%")
print(f"Accuracy with errors found by Confident Learning removed: {round(acc_clean*100, 1)}%")

# Compute reduction in error.
err = ((1-acc_original)-(1-acc_clean))/(1-acc_original)
print(f"Reduction in error: {round(err*100,1)}%")

This proposal ensures that the training process avoids rows with problematic labels, but the predictions still cover the entire dataset, allowing for a fair comparison between acc_original and acc_clean.

Thank you for considering this feedback. I'm looking forward to any thoughts or clarifications on this matter.

Best regards,

RomainBdt

Already labeled data is labeled again in every iteration in "Growing Datasets"

Hi, I'm just doing this course out of personal interest. In

def passive_selection(x, labeled, label_func, n):
    candidates = set(range(0, len(x))) - set(labeled)
    labeled = np.concatenate([labeled, random.sample(list(candidates), n)])
    labels = [label_func(example) for example in x[labeled]]
    return labeled, labels

and

def active_selection(x, labeled, label_func, n):
    labels = [label_func(example) for example in x[labeled]]
    candidates = set(np.arange(len(x))) - set(labeled)
    # YOUR CODE HERE
    pass

both functions apply the label_func to all samples in labeled, so already labeled data will always be rel-labeled. Is there a reason for re-labeling already labeled examples in every iteration. It does not really matter since the label_func is O(1) but in practice this would be very bad I assume?

Can't load train embedding for growing datasets lab

I am working on the growing datasets lab.
When I am trying to load to embedding for the train set, I am loading 32% of the data very fast (33 seconds) but gets stuck there every time.
Here is the slow part of the code (from OpenImagesDataset, _load_embeddings_df):

for filepath in wrapped_loader:
    df = pd.read_parquet(filepath)
    index_ls.append(df.index)
    values_ls.append(df['embedding'].tolist())

My RAM and CPU usage isn't too high.

Edit: I have noticed that my runtime restarts every time (probably because I used more than my 16GB of RAM)

Provide some labs about 3d data such as point cloud

Hi,

nice project!
I would like to know if you can provide some labs for point cloud data, thanks!