Comments (10)
This is interesting, thank you. I believe we have never trained on just source tags, so that must be why we have never encountered this error before. Thank you for reporting it.
Let me just setup a little experiment to try and reproduce this, and will report to you when I understand how this could be fixed.
from openkiwi.
I just tried to train Bert and XLM-R based models on GPU with predicting just source tags, and did not encounter this problem. I used
trainer:
main_metric:
- source_tags_MCC
and
system:
model:
outputs:
word_level:
target: false
gaps: false
source: true
Could you share (the relevant parts of) your config? For example, what is main_metric
that you are using?
from openkiwi.
Thank you for your help, I use source_tags_F1_MULT and source_tags_CORRECT as my main metric, and the sentence level tag is not predicted in my experiment. The relevant parts in my config are as following.
outputs:
####################################################
# Output options configure the downstream tasks the
# model will be trained on by adding specific layers
# responsible for transforming decoder features into
# predictions.
word_level:
target: false
gaps: false
source: true
class_weights:
target_tags:
BAD: 3.0
gap_tags:
BAD: 5.0
source_tags:
BAD: 3.0
sentence_level:
hter: false
use_distribution: false
binary: false
n_layers_output: 2
sentence_loss_weight: 1
and
main_metric:
- source_tags_F1_MULT
- source_tags_CORRECT
from openkiwi.
Hi, @daandouwe
I tried to replace the code in callbacks.py like this, and it seems worked now
# mode_dict = {
# 'min': np.less,
# 'max': np.greater,
# 'auto': np.greater if 'acc' in self.monitor else np.less,
# }
mode_dict = {
'min': torch.lt,
'max': torch.gt,
'auto': torch.gt if 'acc' in self.monitor else torch.lt,
}
from openkiwi.
Thanks!
Turns out that the problem is caused by the fact that not all the metric values have been moved to CPU.
Inspecting the values in the dictionary metrics
in https://github.com/Unbabel/OpenKiwi/blob/master/kiwi/training/callbacks.py#L94 tells us the following:
{'F1_BAD': tensor(0.6000),
'F1_OK': tensor(0.6092),
'loss': tensor(205.8955, device='cuda:0'),
'metrics': {'F1_BAD': tensor(0.6000),
'F1_OK': tensor(0.6092),
'source_tags_CORRECT': tensor(0.6047, device='cuda:0'),
'source_tags_F1_MULT': 0.36551724137931035,
'source_tags_F1_MULT+source_tags_CORRECT': tensor(0.9702, device='cuda:0'),
'source_tags_MCC': tensor(0.2713, dtype=torch.float64)},
'source_tags_CORRECT': tensor(0.6047, device='cuda:0'),
'source_tags_F1_MULT': 0.36551724137931035,
'source_tags_F1_MULT+source_tags_CORRECT': tensor(0.9702, device='cuda:0'),
'source_tags_MCC': tensor(0.2713, dtype=torch.float64),
'val_F1_BAD': tensor(0.4514),
'val_F1_OK': tensor(0.6182),
'val_loss': tensor(141.6612, device='cuda:0'),
'val_loss_source_tags': tensor(141.6612, device='cuda:0'),
'val_source_tags_CORRECT': tensor(0.5497, device='cuda:0'),
'val_source_tags_F1_MULT': 0.27903935726135765,
'val_source_tags_F1_MULT+source_tags_CORRECT': tensor(0.8288, device='cuda:0'),
'val_source_tags_MCC': tensor(0.1847, dtype=torch.float64)}
This is also the reason that main metric source_tags_MCC
works but source_tags_CORRECT
does not. (I believe this flew under the radar because it seems to be the case for the lesser-used metrics).
We will need to solve this by making sure all the metrics return torch tensors that have been moved to CPU. Preferably, all the values returned by metrics will actually just be python floats.
We will try and fix this in a PR.
from openkiwi.
Yes, Thank you for your help! And to replace the fuctions of numpy with the operation of pytorch, maybe another solution?
from openkiwi.
Thanks!
Turns out that the problem is caused by the fact that not all the metric values have been moved to CPU.
Inspecting the values in the dictionary
metrics
in https://github.com/Unbabel/OpenKiwi/blob/master/kiwi/training/callbacks.py#L94 tells us the following:{'F1_BAD': tensor(0.6000), 'F1_OK': tensor(0.6092), 'loss': tensor(205.8955, device='cuda:0'), 'metrics': {'F1_BAD': tensor(0.6000), 'F1_OK': tensor(0.6092), 'source_tags_CORRECT': tensor(0.6047, device='cuda:0'), 'source_tags_F1_MULT': 0.36551724137931035, 'source_tags_F1_MULT+source_tags_CORRECT': tensor(0.9702, device='cuda:0'), 'source_tags_MCC': tensor(0.2713, dtype=torch.float64)}, 'source_tags_CORRECT': tensor(0.6047, device='cuda:0'), 'source_tags_F1_MULT': 0.36551724137931035, 'source_tags_F1_MULT+source_tags_CORRECT': tensor(0.9702, device='cuda:0'), 'source_tags_MCC': tensor(0.2713, dtype=torch.float64), 'val_F1_BAD': tensor(0.4514), 'val_F1_OK': tensor(0.6182), 'val_loss': tensor(141.6612, device='cuda:0'), 'val_loss_source_tags': tensor(141.6612, device='cuda:0'), 'val_source_tags_CORRECT': tensor(0.5497, device='cuda:0'), 'val_source_tags_F1_MULT': 0.27903935726135765, 'val_source_tags_F1_MULT+source_tags_CORRECT': tensor(0.8288, device='cuda:0'), 'val_source_tags_MCC': tensor(0.1847, dtype=torch.float64)}This is also the reason that main metric
source_tags_MCC
works butsource_tags_CORRECT
does not. (I believe this flew under the radar because it seems to be the case for the lesser-used metrics).We will need to solve this by making sure all the metrics return torch tensors that have been moved to CPU. Preferably, all the values returned by metrics will actually just be python floats.
We will try and fix this in a PR.
Yeah, that's more reasonable. Thank you!
from openkiwi.
maybe another solution?
You could edit
current = metrics.get(self.monitor)
if self.monitor_op(current - self.min_delta, self.best):
to
current = metrics.get(self.monitor)
if not isinstance(current, float):
current = current.cpu().item()
if self.monitor_op(current - self.min_delta, self.best):
in https://github.com/Unbabel/OpenKiwi/blob/master/kiwi/training/callbacks.py#L96 (in your local kiwi path).
I tried this and it worked, but of course it's not as nice as dealing with the problem at the root ;). We'll keep you updated on that.
from openkiwi.
maybe another solution?
You could edit
current = metrics.get(self.monitor) if self.monitor_op(current - self.min_delta, self.best):to
current = metrics.get(self.monitor) if not isinstance(current, float): current = current.cpu().item() if self.monitor_op(current - self.min_delta, self.best):in https://github.com/Unbabel/OpenKiwi/blob/master/kiwi/training/callbacks.py#L96 (in your local kiwi path).
I tried this and it worked, but of course it's not at nice and dealing with the problem at the root ;). We'll keep you updated on that.
OK, thanks for your help, now it's woking.
from openkiwi.
Hey @iamhere1,
I'm glad it's working, closing this issue due to inactivity.
from openkiwi.
Related Issues (20)
- TypeError: cannot unpack non-iterable NoneType object HOT 1
- The prediction process is not complete by Predictor Estimator. HOT 5
- OpenKiwi always download the tokenizer files for XLMRoberta even if a local path is configured. HOT 2
- Do openKiwi have confident score? HOT 1
- Error Pre-Training Predictor: "model -> encoder -> encode_source extra fields not permitted (type=value_error.extra)" HOT 1
- some confusions
- pkgutil.iter_modules() error: 'PosixPath' object has no attribute 'startswith'
- Got exception when import kiwi
- Seems that maximum token support for a sentence is 512?
- PicklingError: Can't pickle <class 'kiwi.data.encoders.wmt_qe_data_encoder.InputFields[PositiveInt]'>: attribute lookup InputFields[PositiveInt] on kiwi.data.encoders.wmt_qe_data_encoder failed HOT 2
- Do you need to tokenize your data when using a BERT/ROBERTA model?
- Pretrain config file
- What are source_pos and target_pos in the train_config.yaml?
- Why does it need "--model" paramter when I give a specific config? HOT 2
- What languages do the OpenKiwi support?
- some problems about data without alignments HOT 11
- I suppose that the code comment should be remove. HOT 2
- Error at Predictor Training: "Predictor is not a subclass of QESystem" HOT 2
- OSError: Can't load weights for 'xlm-roberta-base'. HOT 16
- open cannot unpack non-iterable NoneType object HOT 16
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openkiwi.