ductuantruong/mfa_conformer_sv

mfa_conformer_sv's Issues

About train loss

Hello, I noticed that you achieved an impressive EER of 0.72%. I attempted training with similar hyperparameters on a single Nvidia 4090, but my accuracy was only 0.84 with a loss of approximately 0.5. I would appreciate it if you could kindly share the value of your loss after training convergence.

Runtime error: Tensor unequal size

Hello, I am getting this error when beginning to train the model. I have kept the same hyperparameter values except 2,
batch_size = 200, as given in the one you've referred (I tried 360 as well, but I am still getting the same error)
num_classes = 7293, I am using VoxCeleb 1, 2 and the SITW dataset as well, so the extra classes.

I am using your code since some of the libraries from the referred repository have been discontinued.

Epoch 0:   0%|                      | 0/5460 [00:00<00:00, 8338.58it/s]Traceback (most recent call last):
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1045, in _run_train
    self.fit_loop.run()
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 200, in advance    
    epoch_output = self.epoch_loop.run(train_dataloader)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 118, in advance
    _, (batch, is_last) = next(dataloader_iter)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/profiler/base.py", line 104, in profile_iterable
    value = next(iterator)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 625, in prefetch_iterator
    last = next(it)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 546, in __next__
    return self.request_next_batch(self.loader_iters)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 574, in request_next_batch
    return apply_to_collection(loader_iters, Iterator, next_fn)        
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 96, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 561, in next_fn    batch = next(iterator)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__        
    data = self._next_data()
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data     
    return self._process_data(data)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data  
    data.reraise()
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.      
Original Traceback (most recent call last):
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop 
    data = fetcher.fetch(index)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 84, in default_collate
    return [default_collate(samples) for samples in transposed]        
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 84, in <listcomp>   
    return [default_collate(samples) for samples in transposed]        
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 56, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [48000] at entry 0 and [48000, 2] at entry 1


During handling of the above exception, another exception occurred:    

Traceback (most recent call last):
  File "/data1/users/farmaans/mfa_2/mfa_conformer_sv/main.py", line 227, in <module>
    cli_main()
  File "/data1/users/farmaans/mfa_2/mfa_conformer_sv/main.py", line 223, in cli_main
    trainer.fit(model, datamodule=dm)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 553, in fit       
    self._run(model)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 918, in _run      
    self._dispatch()
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 986, in _dispatch 
    self.accelerator.start_training(self)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 92, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in start_training
    self._results = trainer.run_stage()
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 996, in run_stage 
    return self._run_train()
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1058, in _run_train
    self.training_type_plugin.reconciliate_processes(traceback.format_exc())
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 453, in 
reconciliate_processes
    raise DeadlockDetectedException(f"DeadLock detected from rank: {self.global_rank} \n {trace}")
pytorch_lightning.utilities.exceptions.DeadlockDetectedException: DeadLock detected from rank: 1
 Traceback (most recent call last):
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1045, in _run_train
    self.fit_loop.run()
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 200, in advance    
    epoch_output = self.epoch_loop.run(train_dataloader)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 118, in advance
    _, (batch, is_last) = next(dataloader_iter)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/profiler/base.py", line 104, in profile_iterable
    value = next(iterator)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 625, in prefetch_iterator
    last = next(it)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 546, in __next__
    return self.request_next_batch(self.loader_iters)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 574, in request_next_batch
    return apply_to_collection(loader_iters, Iterator, next_fn)        
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 96, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 561, in next_fn    batch = next(iterator)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__        
    data = self._next_data()
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data     
    return self._process_data(data)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data  
    data.reraise()
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.      
Original Traceback (most recent call last):
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop 
    data = fetcher.fetch(index)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 84, in default_collate
    return [default_collate(samples) for samples in transposed]        
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 84, in <listcomp>   
    return [default_collate(samples) for samples in transposed]        
  File "/data1/users/farmaans/mfa_conformer/mfa_env/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 56, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [48000] at entry 0 and [48000, 2] at entry 1


start.sh: line 42: 207482 Killed                  python3 main.py --batch_size 200 --num_workers 40 --max_epochs 30 --embedding_dim $embedding_dim --save_dir $save_dir --encoder_name $encoder_name --train_csv_path $train_csv_path --learning_rate 0.001 --encoder_name ${encoder_name} --num_classes $num_classes --trial_path $trial_path --loss_name $loss_name --num_blocks $num_blocks --step_size 4 --gamma 0.5 --weight_decay 0.0000001 --input_layer $input_layer --pos_enc_layer_type $pos_enc_layer_type