Cuda out of memory error for another dataset for a 45 Gb GPU

[yaswanth-iitkgp]

Cuda out of memory error for another dataset for a 45 Gb GPU

[GanjinZero]

What is your batch size; att_dim; entity type count?
please try batch size=1, lower your att_dim to 192/128/64.
If your entity type is too much (over 50), it is hard to use triaffine based model.

[yaswanth-iitkgp]

Thanks a lot for your quick reply,

I used the same batch size i.e 1. entity type count is 86. I used att_dim = 320 (same as yours).
I'll check reducing it down. Because my entity type count is 86, I cannot use this model at all ?? or is there any other recommendation ?

[GanjinZero]

Maybe try att_dim=128/64 and apply fp16/bf16 for training.

[yaswanth-iitkgp]

sure I'll try with 128 att_dim, and how to apply fp16/bf16 ?

[GanjinZero]

Please refer to this https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision

Our code does not implement it

[yaswanth-iitkgp]

thanks for that. I am able to train the model with att_dim = 128 (consuming only 13gb) , 192(also requires more than 45gb GPU). I haven't implemented that fp16/bf16 assuming it is only for improving performance of gpu.

My dataset is a low resourse dataset so it doesn't have pos, ltokens, rtokens and relations.
can I know if it can be a cause for reduction in fscore ?? I still got a decent score of 65

[GanjinZero]

Without pos Lorien rtoken will reduce your performance, for ace dateset it will downgrade 1-2 f point approximately.

[yaswanth-iitkgp]

Thank you so much for your quick responses and your help. I really appreciate it. I just have 1 last question , does the size of att_dim affect the f1score ?? or is it just for performance of GPU ?

[GanjinZero]

I think it does effect F1, while we do not experiment it.

[yaswanth-iitkgp]

ok thank you very much for you informative answers and help