lyutyuh / asp Goto Github PK
View Code? Open in Web Editor NEWPyTorch implementation and pre-trained models for ASP - Autoregressive Structured Prediction with Language Models, EMNLP 22. https://arxiv.org/pdf/2210.14698.pdf
License: MIT License
PyTorch implementation and pre-trained models for ASP - Autoregressive Structured Prediction with Language Models, EMNLP 22. https://arxiv.org/pdf/2210.14698.pdf
License: MIT License
I want to pass a text file and get the clusters of coreference.
For example,
Input: My name is Sarah and I live in London. I go to the office everyday with my car.
Output: [[My, I, I, my]]
I don't know how to solve this issue. Can you please give me an idea?
Hello,
There is a missing requirements.txt for pip install
Hello, I try to understand what's going on here in the model. So for the NER task, basically the input of the model consists these:
'input_ids',
'input_mask',
'to_copy_ids',
'target_ids',
'target_mask',
'action_labels',
'ent_indices',
'ent_types'
But seems like the model doesn't use 'ent_indices' and 'ent_types' during training. And for 'action_labels', it will always create a cumsum for 0 to (seq_len-1) no matter what 'action_labels' really is. So we got the brackets and [copy] inside 'target_ids', I could understand that but how could the model identify the entity type in NER task?
Thank you
How much GPU memory is required for model training? If the GPU memory is not enough, how can model parameters be optimized (batch_size has been set to 1)?
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)
On running the evaluations as mentioned here: https://github.com/lyutyuh/ASP/tree/master?tab=readme-ov-file#pre-trained-models
python evaluate_ere.py flant5_base_conll04 tliu/asp-re-flan-t5-base 0
It errors out as: FileNotFoundError
for /data/conll04_ere/train_dev.t5-small.jsonlines
Following is the stacktrace:
(asp) ➜ ASP git:(master) ✗ python evaluate_ere.py flant5_base_conll04 tliu/asp-re-flan-t5-base 0
02/26/2024 10:28:38 - INFO - util.func - Running experiment: flant5_base_conll04
02/26/2024 10:28:38 - INFO - util.func - task = "ere"
dataset = "conll04"
data_dir = "/home/kushwaha/Projects/ASP/data/conll04_ere/"
model_dir = "/home/kushwaha/Projects/ASP/data/conll04_ere/"
log_root = "/home/kushwaha/Projects/ASP/data/conll04_ere/"
max_segment_len = 256
use_amp = true
optimizer = "adamw"
plm_learning_rate = 5e-05
task_learning_rate = 0.0001
plm_scheduler = "linear_with_warmup"
task_scheduler = "linear_with_warmup"
warmup_ratio = 0.05
adam_eps = 1e-08
adam_weight_decay = 0.1
init_std = 0.02
max_grad_norm = 1
batch_size = 8
gradient_accumulation_steps = 1
num_epochs = 200
activation = "relu"
dropout_rate = 0.3
feature_emb_size = 20
hidden_size = 1500
num_typing_classes = 4
num_linking_classes = 5
beam_size = 1
eval_frequency = 500
report_frequency = 20
plm_tokenizer_name = "t5-small"
plm_pretrained_name_or_path = "google/flan-t5-base"
log_dir = "/home/kushwaha/Projects/ASP/data/conll04_ere/flant5_base_conll04"
tb_dir = "/home/kushwaha/Projects/ASP/data/conll04_ere/tensorboard"
02/26/2024 10:28:38 - INFO - /home/kushwaha/Projects/ASP/util/runner.py - Log file path: /home/kushwaha/Projects/ASP/data/conll04_ere/flant5_base_conll04/log_Feb26_10-28-38.txt
02/26/2024 10:28:39 - INFO - util.tensorize_ere - Tensorizing examples from /home/kushwaha/Projects/ASP/data/conll04_ere/train_dev.t5-small.jsonlines; results will be cached in /home/kushwaha/Projects/ASP/data/conll04_ere/cached.tensors.t5-small.bin
Traceback (most recent call last):
File "evaluate_ere.py", line 24, in <module>
evaluate(config_name, gpu_id, saved_suffix)
File "evaluate_ere.py", line 7, in evaluate
runner = ERERunner(
File "/home/kushwaha/Projects/ASP/util/runner.py", line 76, in __init__
self.data = EREDataProcessor(self.config)
File "/home/kushwaha/Projects/ASP/util/tensorize_ere.py", line 61, in __init__
with open(path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/kushwaha/Projects/ASP/data/conll04_ere/train_dev.t5-small.jsonlines'
Questions:
12/12/2022 16:09:12 - INFO - util.func - Running experiment: t5_base
Traceback (most recent call last):
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_parser.py", line 538, in _resolve_variable
return True, config.get(variable)
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_tree.py", line 236, in get
return self._get(ConfigTree.parse_key(key), 0, default)
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_tree.py", line 176, in _get
raise ConfigMissingException(
pyhocon.exceptions.ConfigMissingException: 'No configuration setting found for key ASP'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:/svn/ASPCoref/run_ner.py", line 94, in
runner = NERRunner(
File "D:\svn\ASPCoref\util\runner.py", line 50, in init
self.config = util.initialize_config(config_name, config_file=config_file)
File "D:\svn\ASPCoref\util\func.py", line 24, in initialize_config
config = pyhocon.ConfigFactory.parse_file(join("./", config_file))[config_name]
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_parser.py", line 152, in parse_file
return cls.parse_string(content, os.path.dirname(filename), resolve, unresolved_value)
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_parser.py", line 202, in parse_string
return ConfigParser().parse(content, basedir, resolve, unresolved_value)
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_parser.py", line 521, in parse
has_unresolved = cls.resolve_substitutions(config, allow_unresolved)
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_parser.py", line 703, in resolve_substitutions
is_optional_resolved, resolved_value = cls._resolve_variable(config, substitution)
File "C:\Users\Administrator\AppData\Local\conda\conda\envs\asp\lib\site-packages\pyhocon\config_parser.py", line 547, in _resolve_variable
raise ConfigSubstitutionException(
pyhocon.exceptions.ConfigSubstitutionException: Cannot resolve variable ${ASP} (line: 5, col: 14)
That is a really interesting idea proposed. Thanks for the release of the codebase. Can you please tell me when the pre-trained models would be released?
On
python ./run_ere.py t0_3b_conll04 0
06/04/2023 20:04:34 - INFO - util.func - Running experiment: t0_3b_conll04
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_parser.py", line 478, in _resolve_variable
return True, config.get(variable)
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_tree.py", line 236, in get
return self._get(ConfigTree.parse_key(key), 0, default)
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_tree.py", line 176, in _get
raise ConfigMissingException(
pyhocon.exceptions.ConfigMissingException: 'No configuration setting found for key t5_3b_ace05_wu08'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jovyan/au/ASP/./run_ere.py", line 99, in
runner = ERERunner(
File "/home/jovyan/au/ASP/util/runner.py", line 50, in init
self.config = util.initialize_config(config_name, config_file=config_file)
File "/home/jovyan/au/ASP/util/func.py", line 24, in initialize_config
config = pyhocon.ConfigFactory.parse_file(join("./", config_file))[config_name]
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_parser.py", line 142, in parse_file
return cls.parse_string(content, os.path.dirname(filename), resolve, unresolved_value)
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_parser.py", line 192, in parse_string
return ConfigParser().parse(content, basedir, resolve, unresolved_value)
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_parser.py", line 460, in parse
has_unresolved = cls.resolve_substitutions(config, allow_unresolved)
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_parser.py", line 646, in resolve_substitutions
is_optional_resolved, resolved_value = cls._resolve_variable(config, substitution)
File "/opt/conda/lib/python3.10/site-packages/pyhocon/config_parser.py", line 487, in _resolve_variable
raise ConfigSubstitutionException(
pyhocon.exceptions.ConfigSubstitutionException: Cannot resolve variable ${t5_3b_ace05_wu08} (line: 191, col: 20)
t5_3b_ace05_wu08
to ere.confflant5_xxl_ace05
subconfig${t5_3b_ace05_wu08}
to ${t5_3b_ace05}
(link to another subconfig)Take conll04 for example
'predicted_relations':[((22, 22), (38, 39), 7, 0, 2)]
what does 7 mean?it has only 5 relations.
Thanks for the code! It's a really interesting idea!
However, I'm trying to run the NER pipeline as-is using the given commands but I'm unable to do so due to some issues installing apex. Would it be possible to provide an option to run the code without using apex?
# Evaluate
if self.scheduler._step_count % self.config['eval_frequency'] == 0:
logger.info('Dev')
f1, _ = self.evaluate(
model, examples_dev, stored_info, self.scheduler._step_count
)
logger.info('Test')
f1_test = 0. # It is always zero
if f1 > max_f1:
max_f1 = max(max_f1, f1)
max_f1_test = 0.
self.save_model_checkpoint(
model, self.optimizer, self.scheduler, self.scheduler._step_count, epo
)
logger.info('Eval max f1: %.2f' % max_f1)
logger.info('Test max f1: %.2f' % max_f1_test)
start_time = time.time()
08/18/2023 05:15:37 - INFO - /mnt/ssd_mnt/pyj/ASP/util/runner.py - Log file path: /mnt/ssd_mnt/pyj/ASP/data/ontonotes_coref/t5_base/log_Aug18_05-15-37.txt
08/18/2023 05:15:37 - INFO - util.tensorize_coref - Loaded tensorized examples from cache: /mnt/ssd_mnt/pyj/ASP/data/ontonotes_coref/cached.tensors.t5-small.2048.bin
/opt/conda/envs/asp/lib/python3.8/site-packages/torch/cuda/init.py:546: UserWarning: Can't initialize NVML
warnings.warn("Can't initialize NVML")
Traceback (most recent call last):
File "run_coref.py", line 121, in
model, _ = runner.initialize_model()
File "/mnt/ssd_mnt/pyj/ASP/util/runner.py", line 78, in initialize_model
model = self.model_class_fn(self.config, self.device)
File "/mnt/ssd_mnt/pyj/ASP/models/model_coref.py", line 49, in init
self.model = T5Coref.from_pretrained(
File "/opt/conda/envs/asp/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2228, in from_pretrained
model = cls(config, *model_args, **model_kwargs)
File "/mnt/ssd_mnt/pyj/ASP/models/t5_coref.py", line 62, in init
self.action_head = util.make_ffnn(
File "/mnt/ssd_mnt/pyj/ASP/util/func.py", line 402, in make_ffnn
ffnn = [make_linear(feat_size, hidden_size[0], std=std), ACT2FN[activation], dropout]
File "/mnt/ssd_mnt/pyj/ASP/util/func.py", line 385, in make_linear
linear = nn.Linear(
File "/opt/conda/envs/asp/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 96, in init
self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
RuntimeError: Device index must not be negative
How can I fix it??
Dear Tianyu Liu,
I really like the paper and the idea! And also thank you for releasing the code base!
I am currently working on my master's thesis and I am planning to augment this architecture with knowledge infusion.
While doing so, I encountered an issue with the code to convert the CoNLL03 dataset to the required json structure.
In the tables below, you can see that using your code (denoted eth_asp) does not capture 27 entities over the train, dev and test sets.
Your code does not check for entities at the end of the document -> they are not recognized.
I propose the following changes to your code:
if line == "-DOCSTART- -X- -X- O": # new doc
if doc is not None:
# when extended is not the same as tokens
# mark where to copy from with <extra_id_22> and <extra_id_23>
# E.g.
# Extract entities such as apple, orange, lemon <extra_id_22> Give me a mango . <extra_id_23>
# See ace05_to_json.py for example of extending the input
# FIX: missing entities <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
if start is not None:
doc['entities'].append({
"type":
current_type,
"start":
start,
"end":
idx if idx > start else idx + 1
})
# FIX: missing entities >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
doc["extended"] = doc["tokens"]
dataset.append(doc)
doc = {
"tokens": [], # list of tokens for the model to copy from
"extended":
[], # list of input tokens. Prompts, instructions, etc. go here
"entities": [
] # list of dict:{"type": type, "start": start, "end": end}, format: [start, end)
}
idx, start = -1, None
continue
Best regards,
Robin
Hello,
Seems like for NER it's possible to extract entities with the same right boundary. However, for ERE, is it possible to distinguish the subject or object's correct entity span if there are other entities which have the same right boundaries?
Thank you
Sorry but I didn't find the 'conll03_to_json.py'.
Could you please check that?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.