A question about ActiveLearningDataset.

[baal-org]

A question about ActiveLearningDataset.

[Dref360]

Hello,

Would your class Data have a field named label?

This is definitely a bug on our side, we should check that self._dataset.label is a Callable.

Thank you for raising this issue.

I'll explain the behavior here for other users and I would add this to our FAQ as well.

In Baal, we allow both a research mode and a production mode.

• Research mode
  • Assume that you already have the labels, so we ignore the value even if it is not None.
• Production mode
  • To label something, we assume that the inner dataset has a method named label and we forward the value to it.

We do have a tutorial on how to use Baal in Production. It uses our class FileDataset which does have a label method.

https://baal.readthedocs.io/en/latest/notebooks/baal_prod_cls.html

Thanks again for reporting this issue!

[thu-wangz17]

Oh,sorry.I forgot to copy the class Data above.Thank you for your advice.This is the original class Data:

class Data(Dataset):
    def __init__(self, feature, label=None):
        self.feature = feature
        self.label = label
    
    def __getitem__(self, index):
        if self.label:
            return self.feature[index], self.label[index]
        else:
            return self.feature[index]
    
    def __len__(self):
        return len(self.feature)

However I'm still confused about how to make the class Data have a field named label?Does you mean that I should define a label method for the unlabelled data to query the label?

class Data(Dataset):
    def __init__(self, feature):
        self.feature = feature
    
    def __getitem__(self, index):
        return self.feature[index]
    
    def __len__(self):
        return len(self.feature)

    def label(self, index):
        return self.feature[index].sum()

when I use pool.label(index=4),it still has a problem:

c:\users\anaconda3\lib\site-packages\baal\active\dataset.py:170: UserWarning: The dataset is able to label data, but no label was provided.
                                 The dataset will be unchanged from this action!
                                 If this is a research setting, please set the
                                  `ActiveLearningDataset.can_label` to `False`.

  """, UserWarning)

Could you give me an example?Thank you very much.

[Dref360]

yes a label method sorry.

The warning you see is normal. If it gets too verbose, we should suppress it.

If the labels are already available then you don't need this method.

But in the case of a live application you would get something like this:

# Some definitions
your_heuristic = BALD()
pool = active_dataset.pool
your_predictions = ModelWrapper.predict_on_dataset(pool, ...)
# The shape of `your_predictions` is [len(pool), n_classes, ..., iterations]
# Get the next batch of samples to label. Note: These indices are according to the pool.
ranks = your_heuristic(your_predictions)

# Now let's ask a human to label those samples.
labels = ask_a_human(ranks, pool)

# To edit the dataset labels, you can now add those labels to your dataset. Still, the indices are according to the pool.
active_dataset.label(ranks, labels)

See how we make the label method in FileDataset here

[Dref360]

If you don't know the label, you should just return a dummy value for the labels.

I would propose something like this:

class Data(Dataset):
    def __init__(self, feature, label=None):
        self.feature = feature
        self.lbl = label or [-1] * len(feature)
    
    def __getitem__(self, index):
        return self.feature[index], self.lbl[index]
    
    def __len__(self):
        return len(self.feature)

	def label(self, index, value):
		self.lbl[index] = value

[Dref360]

I'm currently working on our FAQ. here is a section that I would like to add on the split. If there is something missing, please let me know and I'll add it.

How can I specify that a label is missing and how to label it.

The source of truth for what is labelled is the ActiveLearningDataset._labelled array.
This means that we will never train on a sample if it is not labelled according to this array.
This array determines the split between the labelled and unlabelled datasets.

# Let ds = D, the entire dataset with labelled/unlabelled data.
ds = YourDataset()
al_dataset = ActiveLearningDataset(ds, ...)
# For convenience, let's label 10 samples at random.
# But you can provide the `labelled` array to ActiveLearningDataset
# if you already have labels.
al_dataset.label_randomly(10)
pool = al_dataset.pool

From a rigorous point of view: $D = ds$, $D_L=al_dataset$ and $D_U = D \setminus D_L = pool$.
Then, we train our model on $D_L$ and compute the uncertainty on $D_U$. The most uncertain samples are labelled and added to $D_L$, removed from $D_U$.

Let a method query_human performs the annotations, we can label our dataset using indices relative to $D_U$. This assumes that your dataset class YourDataset has a method named label which has the following definition: def label(self, idx, value) where we give the label for indice idx. There the indice is not relative to the pool, so you don't have to worry about it.

[thu-wangz17]

That's very clear.Thank you very much.