lyutyuh/asp

PyTorch implementation and pre-trained models for ASP - Autoregressive Structured Prediction with Language Models, EMNLP 22. https://arxiv.org/pdf/2210.14698.pdf

asp's Issues

Missing test evaluation

Missing code for test evaluation, only dev evaluation exists.

# Evaluate
                    if self.scheduler._step_count % self.config['eval_frequency'] == 0:
                        logger.info('Dev')

                        f1, _ = self.evaluate(
                            model, examples_dev, stored_info, self.scheduler._step_count
                        )
                        logger.info('Test')
                        f1_test = 0. # It is always zero
                        if f1 > max_f1:
                            max_f1 = max(max_f1, f1)
                            max_f1_test = 0. 
                            self.save_model_checkpoint(
                                model, self.optimizer, self.scheduler, self.scheduler._step_count, epo
                            )

                        logger.info('Eval max f1: %.2f' % max_f1)
                        logger.info('Test max f1: %.2f' % max_f1_test)
                        start_time = time.time()

Model inputs

Hello, I try to understand what's going on here in the model. So for the NER task, basically the input of the model consists these:

'input_ids',
'input_mask',
'to_copy_ids',
'target_ids',
'target_mask',
'action_labels',
'ent_indices',
'ent_types'

But seems like the model doesn't use 'ent_indices' and 'ent_types' during training. And for 'action_labels', it will always create a cumsum for 0 to (seq_len-1) no matter what 'action_labels' really is. So we got the brackets and [copy] inside 'target_ids', I could understand that but how could the model identify the entity type in NER task?
Thank you

About entity's right boundary in ERE

Hello,

Seems like for NER it's possible to extract entities with the same right boundary. However, for ERE, is it possible to distinguish the subject or object's correct entity span if there are other entities which have the same right boundaries?

Thank you

How to understand the predicted relations

Take conll04 for example
'predicted_relations':[((22, 22), (38, 39), 7, 0, 2)]
what does 7 mean?it has only 5 relations.

GPU Memory

How much GPU memory is required for model training? If the GPU memory is not enough, how can model parameters be optimized (batch_size has been set to 1)?

Missing entities on data preparation with conll03_to_json.py

Dear Tianyu Liu,

I really like the paper and the idea! And also thank you for releasing the code base!
I am currently working on my master's thesis and I am planning to augment this architecture with knowledge infusion.

While doing so, I encountered an issue with the code to convert the CoNLL03 dataset to the required json structure.
In the tables below, you can see that using your code (denoted eth_asp) does not capture 27 entities over the train, dev and test sets.

Your code does not check for entities at the end of the document -> they are not recognized.

I propose the following changes to your code:

          if line == "-DOCSTART- -X- -X- O":  # new doc
                if doc is not None:
                    # when extended is not the same as tokens
                    # mark where to copy from with <extra_id_22> and <extra_id_23>
                    # E.g.
                    # Extract entities such as apple, orange, lemon <extra_id_22> Give me a mango . <extra_id_23>
                    # See ace05_to_json.py for example of extending the input

                    # FIX: missing entities  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                    if start is not None:
                        doc['entities'].append({
                            "type":
                            current_type,
                            "start":
                            start,
                            "end":
                            idx if idx > start else idx + 1
                        })
                    # FIX: missing entities >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
                    
                    doc["extended"] = doc["tokens"]
                    dataset.append(doc)
                doc = {
                    "tokens": [],  # list of tokens for the model to copy from
                    "extended":
                    [],  # list of input tokens. Prompts, instructions, etc. go here
                    "entities": [
                    ]  # list of dict:{"type": type, "start": start, "end": end}, format: [start, end)
                }
                idx, start = -1, None
                continue

Best regards,
Robin

run_ner.py error：Is this error related to the path of Pretrained LM?

12/12/2022 16:09:12 - INFO - util.func - Running experiment: t5_base
Traceback (most recent call last):
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_parser.py", line 538, in _resolve_variable
return True, config.get(variable)
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_tree.py", line 236, in get
return self._get(ConfigTree.parse_key(key), 0, default)
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_tree.py", line 176, in _get
raise ConfigMissingException(
pyhocon.exceptions.ConfigMissingException: 'No configuration setting found for key ASP'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:/svn/ASPCoref/run_ner.py", line 94, in
runner = NERRunner(
File "D:\svn\ASPCoref\util\runner.py", line 50, in init
self.config = util.initialize_config(config_name, config_file=config_file)
File "D:\svn\ASPCoref\util\func.py", line 24, in initialize_config
config = pyhocon.ConfigFactory.parse_file(join("./", config_file))[config_name]
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_parser.py", line 152, in parse_file
return cls.parse_string(content, os.path.dirname(filename), resolve, unresolved_value)
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_parser.py", line 202, in parse_string
return ConfigParser().parse(content, basedir, resolve, unresolved_value)
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_parser.py", line 521, in parse
has_unresolved = cls.resolve_substitutions(config, allow_unresolved)
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_parser.py", line 703, in resolve_substitutions
is_optional_resolved, resolved_value = cls._resolve_variable(config, substitution)
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_parser.py", line 547, in _resolve_variable
raise ConfigSubstitutionException(
pyhocon.exceptions.ConfigSubstitutionException: Cannot resolve variable ${ASP} (line: 5, col: 14)

How can I make prediction from a text file

I want to pass a text file and get the clusters of coreference.
For example,

Input: My name is Sarah and I live in London. I go to the office everyday with my car.
Output: [[My, I, I, my]]

I don't know how to solve this issue. Can you please give me an idea?

Issue with installing Apex. Any option to run code without it?

Thanks for the code! It's a really interesting idea!

However, I'm trying to run the NER pipeline as-is using the given commands but I'm unable to do so due to some issues installing apex. Would it be possible to provide an option to run the code without using apex?

Upload the finetuned model please

missing requirements.txt

Hello,

There is a missing requirements.txt for pip install

pyhocon: Cannot resolve variable ${t5_3b_ace05_wu08}

ere.conf missing subconfig t5_3b_ace05_wu08

On
python ./run_ere.py t0_3b_conll04 0

06/04/2023 20:04:34 - INFO - util.func - Running experiment: t0_3b_conll04
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_parser.py", line 478, in _resolve_variable
return True, config.get(variable)
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_tree.py", line 236, in get
return self._get(ConfigTree.parse_key(key), 0, default)
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_tree.py", line 176, in _get
raise ConfigMissingException(
pyhocon.exceptions.ConfigMissingException: 'No configuration setting found for key t5_3b_ace05_wu08'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/jovyan/au/ASP/./run_ere.py", line 99, in
runner = ERERunner(
File "/home/jovyan/au/ASP/util/runner.py", line 50, in init
self.config = util.initialize_config(config_name, config_file=config_file)
File "/home/jovyan/au/ASP/util/func.py", line 24, in initialize_config
config = pyhocon.ConfigFactory.parse_file(join("./", config_file))[config_name]
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_parser.py", line 142, in parse_file
return cls.parse_string(content, os.path.dirname(filename), resolve, unresolved_value)
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_parser.py", line 192, in parse_string
return ConfigParser().parse(content, basedir, resolve, unresolved_value)
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_parser.py", line 460, in parse
has_unresolved = cls.resolve_substitutions(config, allow_unresolved)
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_parser.py", line 646, in resolve_substitutions
is_optional_resolved, resolved_value = cls._resolve_variable(config, substitution)
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_parser.py", line 487, in _resolve_variable
raise ConfigSubstitutionException(
pyhocon.exceptions.ConfigSubstitutionException: Cannot resolve variable ${t5_3b_ace05_wu08} (line: 191, col: 20)

To fix it

• Add a subconfig t5_3b_ace05_wu08 to ere.conf
• Remove flant5_xxl_ace05 subconfig
• Change ${t5_3b_ace05_wu08} to ${t5_3b_ace05} (link to another subconfig)

Issue in running the evaluation for ERE as per README.md

On running the evaluations as mentioned here: https://github.com/lyutyuh/ASP/tree/master?tab=readme-ov-file#pre-trained-models

python evaluate_ere.py flant5_base_conll04 tliu/asp-re-flan-t5-base 0

It errors out as: FileNotFoundError for /data/conll04_ere/train_dev.t5-small.jsonlines

Following is the stacktrace:

(asp) ➜  ASP git:(master) ✗ python evaluate_ere.py flant5_base_conll04 tliu/asp-re-flan-t5-base 0
02/26/2024 10:28:38 - INFO - util.func - Running experiment: flant5_base_conll04
02/26/2024 10:28:38 - INFO - util.func - task = "ere"
dataset = "conll04"
data_dir = "/home/kushwaha/Projects/ASP/data/conll04_ere/"
model_dir = "/home/kushwaha/Projects/ASP/data/conll04_ere/"
log_root = "/home/kushwaha/Projects/ASP/data/conll04_ere/"
max_segment_len = 256
use_amp = true
optimizer = "adamw"
plm_learning_rate = 5e-05
task_learning_rate = 0.0001
plm_scheduler = "linear_with_warmup"
task_scheduler = "linear_with_warmup"
warmup_ratio = 0.05
adam_eps = 1e-08
adam_weight_decay = 0.1
init_std = 0.02
max_grad_norm = 1
batch_size = 8
gradient_accumulation_steps = 1
num_epochs = 200
activation = "relu"
dropout_rate = 0.3
feature_emb_size = 20
hidden_size = 1500
num_typing_classes = 4
num_linking_classes = 5
beam_size = 1
eval_frequency = 500
report_frequency = 20
plm_tokenizer_name = "t5-small"
plm_pretrained_name_or_path = "google/flan-t5-base"
log_dir = "/home/kushwaha/Projects/ASP/data/conll04_ere/flant5_base_conll04"
tb_dir = "/home/kushwaha/Projects/ASP/data/conll04_ere/tensorboard"
02/26/2024 10:28:38 - INFO - /home/kushwaha/Projects/ASP/util/runner.py - Log file path: /home/kushwaha/Projects/ASP/data/conll04_ere/flant5_base_conll04/log_Feb26_10-28-38.txt
02/26/2024 10:28:39 - INFO - util.tensorize_ere - Tensorizing examples from /home/kushwaha/Projects/ASP/data/conll04_ere/train_dev.t5-small.jsonlines; results will be cached in /home/kushwaha/Projects/ASP/data/conll04_ere/cached.tensors.t5-small.bin
Traceback (most recent call last):
  File "evaluate_ere.py", line 24, in <module>
    evaluate(config_name, gpu_id, saved_suffix)
  File "evaluate_ere.py", line 7, in evaluate
    runner = ERERunner(
  File "/home/kushwaha/Projects/ASP/util/runner.py", line 76, in __init__
    self.data = EREDataProcessor(self.config)
  File "/home/kushwaha/Projects/ASP/util/tensorize_ere.py", line 61, in __init__
    with open(path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/kushwaha/Projects/ASP/data/conll04_ere/train_dev.t5-small.jsonlines'

Questions:

• Where to find or generate train_dev.t5-small.jsonlines file?
• There is a link for pre-trained weights: where to configure it to be used?

RuntimeError: Device index must not be negative

08/18/2023 05:15:37 - INFO - /mnt/ssd_mnt/pyj/ASP/util/runner.py - Log file path: /mnt/ssd_mnt/pyj/ASP/data/ontonotes_coref/t5_base/log_Aug18_05-15-37.txt
08/18/2023 05:15:37 - INFO - util.tensorize_coref - Loaded tensorized examples from cache: /mnt/ssd_mnt/pyj/ASP/data/ontonotes_coref/cached.tensors.t5-small.2048.bin
/opt/conda/envs/asp/lib/python3.8/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "run_coref.py", line 121, in
model, _ = runner.initialize_model()
File "/mnt/ssd_mnt/pyj/ASP/util/runner.py", line 78, in initialize_model
model = self.model_class_fn(self.config, self.device)
File "/mnt/ssd_mnt/pyj/ASP/models/model_coref.py", line 49, in init
self.model = T5Coref.from_pretrained(
File "/opt/conda/envs/asp/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2228, in from_pretrained
model = cls(config, *model_args, **model_kwargs)
File "/mnt/ssd_mnt/pyj/ASP/models/t5_coref.py", line 62, in init
self.action_head = util.make_ffnn(
File "/mnt/ssd_mnt/pyj/ASP/util/func.py", line 402, in make_ffnn
ffnn = [make_linear(feat_size, hidden_size[0], std=std), ACT2FN[activation], dropout]
File "/mnt/ssd_mnt/pyj/ASP/util/func.py", line 385, in make_linear
linear = nn.Linear(
File "/opt/conda/envs/asp/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 96, in init
self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
RuntimeError: Device index must not be negative

How can I fix it??

Pretrained Models

That is a really interesting idea proposed. Thanks for the release of the codebase. Can you please tell me when the pre-trained models would be released?

Missing 'conll03_to_json.py'

Sorry but I didn't find the 'conll03_to_json.py'.
Could you please check that?