lena-voita/description-length-probing

This is a repository with the code for the EMNLP 2020 paper "Information-Theoretic Probing with Minimum Description Length"

description-length-probing's Issues

Error in Jupyter Notebook & which checkpoint to load?

Hi Lena,

I am trying to reproduce your edge probing experiments (Description Length and Random Models). It seems some cells are deleted in this notebook. For example, if I run the notebook sequentially, Cell 24 will report a key error. Would you please make sure the same experiment can be reproduced by running the cells sequentially from top to the bottom? Thanks a lot!

Also, the SRL result generated by the notebook matches that reported in the paper, but why do you load checkpoint 400? 400 validations seem to be much less than 200 epochs, which is claimed in the paper.

Thank you for your help in advance!

Mapping from word sequences to subword sequences

description-length-probing/control_tasks/control_tasks/data.py

Line 325 in 2696af0

def match_tokenized_to_untokenized(self, tokenized_sent, untokenized_sent):

Regarding this function, I found the following error case.
Even though this may be a minor error, just for your information.

from transformers import AutoTokenizer

# preparing an example
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
untokenized_sent = 'pretrained language models prone to learn domain-specific spurious correlations between input and output .'.split()
tokenized_sent = tokenizer.tokenize(tokenizer.cls_token + ' '.join(untokenized_sent) + tokenizer.sep_token)

# exactly the same as `match_tokenized_to_untokenized` for generating `mapping`
mapping = defaultdict(list)
untokenized_sent_index = 0
tokenized_sent_index = 1
while (untokenized_sent_index < len(untokenized_sent) and tokenized_sent_index < len(tokenized_sent)):
    while (tokenized_sent_index + 1 < len(tokenized_sent) and tokenized_sent[tokenized_sent_index + 1].startswith('##')):
        mapping[untokenized_sent_index].append(tokenized_sent_index)
        tokenized_sent_index += 1
    mapping[untokenized_sent_index].append(tokenized_sent_index)
    untokenized_sent_index += 1
    tokenized_sent_index += 1

# verifying the mapping is correct or not
for i in mapping:
    j = mapping[i]
    print(untokenized_sent[i], tokenized_sent[j[0]:j[-1]+1])

Result:

pretrained ['pre', '##train', '##ed']
language ['language']
models ['models']
prone ['prone']
to ['to']
learn ['learn']
**domain-specific ['domain']**
spurious ['-']
correlations ['specific']
between ['spur', '##ious']
input ['correlation', '##s']
and ['between']
output ['input']
. ['and']