psunlpgroup/container

Code for ACL 2022 paper "CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learning"

container's Issues

a problem of Reproduce the results with the same seed

hello, when i run the code,i find that i can't get the same loss or f1 score with fixed seed.
for example i set seed==0,and the f1 result will get 43.25, 43.20, 42.89 etc.
i print the output and find that first finetune round the model get the fixed bert out and loss,then loss.backward()，the second round the bert out or loss will change slightly.
it's so wired that every time rerun the scripts, the result always change even fix the seed.
whether the loss function is too complex and The model accuracy is not enough

A question for benchmarks shown in the paper.

I notice that there is a gap between FewNERD official baselines and the ones that come from the performance reported in your paper. Specifically, in the INTER 5 way 1-2 shot setting, ProtoBERT got 44.44 F1 score in your paper while in the FewNERD official benchmarks, it got just 38.83 F1 score. In fact, not only the ProtoBERT model, but all models in the INTER 5 way 1-2 shot setting have the same issue. We can see that in INTER 5 way 1-2 shot setting, StructShot has a F1 score of 57.33, which even exceeds the performance reported in the INTER 5way 5-10 shot setting. Why is this possible? Are there some misunderstandings or the benchmarks just have some problems?

Sorry, why I did not seen the relevant code about function (6) in the training progress, as you said the algorithm 1 in your paper?

model.train()
batch = tuple(t.to(args.device) for t in batch)
inputs = {"input_ids": batch[0], "attention_mask": batch[1],
"token_type_ids": batch[2], "labels": batch[3],"loss_type":args.training_loss,
"consider_mutual_O": args.consider_mutual_O}

        outputs = model(**inputs)
        loss = outputs[0]

        loss.backward()
        training_loss += loss.item()
        torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm)
        optimizer.step()
        model.zero_grad()
        global_step += 1
            # TODO remove args.save_steps
return global_step, training_loss / global_step if global_step > 0 else 0

What is the purpose of the cnts variable in utils.py?

def nt_xent(loss, num, denom, temperature = 1):

loss = torch.exp(loss/temperature)
cnts = torch.sum(num, dim = 1)
loss_num = torch.sum(loss * num, dim = 1)
loss_denom = torch.sum(loss * denom, dim = 1)
# sanity check
nonzero_indexes = torch.where(cnts > 0)
loss_num, loss_denom, cnts = loss_num[nonzero_indexes], loss_denom[nonzero_indexes], cnts[nonzero_indexes]

loss_final = -torch.log2(loss_num) + torch.log2(loss_denom) + torch.log2(cnts)
return loss_final

can you share test data for domain transfer , thank you

I'm not sure but may I ask some code problems

in line 129, 130,131, 115 of file utils.py

1. for line 115, assert a tuple will return none value and will not report error.

2. in line 129, 130,131, 115, the filtered_embedding/repeated_output_embeddings are repeted by 'len(output_embedding_mu)/filtered_instances_nos' times. So these tensors can not be reshape as (filtered_instances_nos, filtered_instances_nos), I change that to (filtered_instances_nos, -1) and code works.

I'm not realy sure about these problems, could you please help me for that?

https://github.com/psunlpgroup/CONTaiNER/blob/main/src/utils.py#L129

why this problem occurs?

File "D:/Project/CONTaiNER/src/container.py", line 790, in
main()
File "D:/Project/CONTaiNER/src/container.py", line 707, in main
global_step, tr_loss = train(args, train_dataset, model)
File "D:/Project/CONTaiNER/src/container.py", line 142, in train
outputs = model(**inputs)
File "D:\Project\CONTaiNER\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "D:\Project\CONTaiNER\src\utils.py", line 191, in forward
loss = calculate_KL_or_euclidean(self, attention_mask, original_embedding_mu,
File "D:\Project\CONTaiNER\src\utils.py", line 115, in calculate_KL_or_euclidean
assert len(repeated_labels) == (filtered_instances_nos * filtered_instances_nos), "dimension is not of square shape."
AssertionError: dimension is not of square shape.
python-BaseException

sorry to bother you, can you help me?

dataset

May I ask how other datasets are processed?

数据链接失效

I can't get the episode_data.zip through the given link,can you update this?

Dataset Release

Could you release the few-shot datasets in Table 3? Because I find these full datasets are all available publicly. What I need is the same few-shot settings with you. Thanks a lot.

How do I run on Tag Set Extension and Domain Transfer tasks?

• The datasets are required to be in the OntoNotes like NER format as organized in the case of Few-NERD.
  And the paper says:
• We also use this evaluation strategy for these traditional NER datasets.

Does Tag Set Extension and Domain Transfer need to sample Support and Query (not only provided) for testing multiple times?

Thank you for this repository.

Ontonotes dataset related questions

Hello, I found that the size of the Ontonotes dataset I processed is different from the one you posted in. Did you do any data deduplication or other work? Thank you for your reply~

AssertionError: dimension is not of square shape.

Hi,
I am trying to run your code but it is giving me this error:

Iteration:   0%|                                                                                                                       | 0/3110 [00:00<?, ?it/s]
Epoch:   0%|                                                                                                                              | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "src/container.py", line 732, in <module>
    main()
  File "src/container.py", line 649, in main
    global_step, tr_loss = train(args, train_dataset, model)
  File "src/container.py", line 140, in train
    outputs = model(**inputs)
  File "/u/pranjalgupta/.local/share/virtualenvs/CONTaiNER-y1BOzUqP/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/dccstor/pranjalgupta2/CONTaiNER/src/utils.py", line 191, in forward
    loss = calculate_KL_or_euclidean(self, attention_mask, original_embedding_mu,
  File "/dccstor/pranjalgupta2/CONTaiNER/src/utils.py", line 115, in calculate_KL_or_euclidean
    assert len(repeated_labels) == (filtered_instances_nos * filtered_instances_nos), "dimension is not of square shape."
AssertionError: dimension is not of square shape.

Steps to reproduce:

1. Download the dataset using dataset.sh
2. Run process_fewnerd.sh script
3. Uncomment ## training with toy evaluation for sanity check section in exec_container.sh script
4. Run exec_container.sh script with the intra 0 5 5 as command line arguments

Can you please help me in resolving this error? Thanks!

can you share the code about how to sample the support set?