lime-robot/categories-prediction

트랜스포머 기반의 상품 카테고리 분류기

categories-prediction's Issues

1등 솔루션 예제 코드를 돌리다가 아래와 같은 CUDA error: device-side assert triggered 오류가 나 해결 방법이 있나 해서 글 남깁니다.

ubuntu@nipa2021-35521:~/taemin/categories-prediction/code$ python train.py --fold 0
2021-11-22 11:04:25.561968: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Device: cuda
Current cuda device: 0
Count of using GPUs: 1
{'module': 'main', 'learning_rate': 0.0003, 'batch_size': 1024, 'num_workers': 4, 'print_freq': 100, 'start_epoch': 0, 'num_train_epochs': 3, 'warmup_steps': 100, 'max_grad_norm': 10, 'weight_decay': 0.01, 'dropout': 0.2, 'hidden_size': 512, 'intermediate_size': 256, 'nlayers': 2, 'nheads': 8, 'seq_len': 64, 'n_b_cls': 58, 'n_m_cls': 553, 'n_s_cls': 3191, 'n_d_cls': 405, 'vocab_size': 32000, 'img_feat_size': 2048, 'type_vocab_size': 30, 'csv_path': '/home/ubuntu/taemin/categories-prediction/input/processed/train.csv', 'h5_path': '/home/ubuntu/taemin/categories-prediction/input/processed/train_img_feat.h5', 'dict': <attribute 'dict' of 'CFG' objects>, 'weakref': <attribute 'weakref' of 'CFG' objects>, 'doc': None, 'seed': 7}
loading ...
loading ... done
parameters: 24637551
num_train_optimization_steps 19065
use WarmupLinearSchedule ...
initial learning rate:0.0
/opt/conda/conda-bld/pytorch_1634272115665/work/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [11,0,0] Assertion t >= 0 && t < n_classes failed.
/opt/conda/conda-bld/pytorch_1634272115665/work/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [15,0,0] Assertion t >= 0 && t < n_classes failed.
Traceback (most recent call last):
File "train.py", line 514, in
main()
File "train.py", line 233, in main
train_res = train(train_loader, model, optimizer, epoch, scheduler)
File "train.py", line 316, in train
loss.backward()
File "/home/ubuntu/anaconda3/envs/transformers/lib/python3.6/site-packages/torch/_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/ubuntu/anaconda3/envs/transformers/lib/python3.6/site-packages/torch/autograd/init.py", line 156, in backward
allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag
RuntimeError: CUDA error: device-side assert triggered

github에 있는 code에서 따로 수정한 부분은 없었습니다.