Both batches need to have the same number of samples

[torch-points3d]

Both batches need to have the same number of samples

[CCInc]

Closing for now, feel free to reopen if still an issue.

[Stakhan]

Running with different size in input data and now the batch sizes are actually different right before the function call:

x.shape = torch.Size([52090, 3])
y.shape = torch.Size([52089, 3])
x_batch.shape = torch.Size([52090])
y_batch.shape = torch.Size([52089])

And this time, I obviously get the error:

  2%|█▎                                                   | 12/499 [02:43<1:50:48, 13.65s/it, data_loading=0.002, iteration=4.06 , train_acc=38.42, train_loss_seg=1.489, train_macc=17.13, train_miou=9.592)]

[...]

  File "C:\Users\EAL\Anaconda3\envs\testenv6_git_inst\lib\site-packages\torch_points_kernels\torchpoints.py", line 168, in ball_query_partial_dense
    ind, dist = tpcpu.batch_ball_query(x, y, batch_x, batch_y, radius, nsample, mode=0, sorted=sort)
RuntimeError: Both batches need to have the same number of samples.

I would be tempted to close the issue but I can reproduce the first case scenario where the batches are of equal size...

[nicolas-chaulet]

Hey! It could be that the log of the batch size and the actual error are not part of the same batch. Is this happening in a data loader?

[Stakhan]

I see.
The batch size is logged right before function call, directly in https://github.com/nicolas-chaulet/torch-points-kernels/blob/master/torch_points_kernels/torchpoints.py#L167:

                print('==>', x.shape, y.shape, batch_x.shape, batch_y.shape)
                ind, dist = tpcpu.batch_ball_query(x, y, batch_x, batch_y, radius, nsample, mode=0, sorted=sort)

I guess it involves a data loader but I'm not entirely sure. Here is the complete stack trace:

==> torch.Size([3431, 3]) torch.Size([3431, 3]) torch.Size([3431]) torch.Size([3431])
==> torch.Size([3431, 3]) torch.Size([3431, 3]) torch.Size([3431]) torch.Size([3431])
==> torch.Size([3431, 3]) torch.Size([3431, 3]) torch.Size([3431]) torch.Size([3431])
 24%|████████████▊                                        | 915/3772 [09:09<24:55,  1.91it/s, data_loading=0.002, iteration=0.282, train_acc=46.87, train_loss_seg=1.340, train_macc=30.17, train_miou=11.29)]
==> torch.Size([2786, 3]) torch.Size([2786, 3]) torch.Size([2786]) torch.Size([2786])
==> torch.Size([2786, 3]) torch.Size([2786, 3]) torch.Size([2786]) torch.Size([2786])
==> torch.Size([2786, 3]) torch.Size([2786, 3]) torch.Size([2786]) torch.Size([2786])
 24%|████████████▊                                        | 915/3772 [09:09<28:36,  1.66it/s, data_loading=0.002, iteration=0.282, train_acc=46.87, train_loss_seg=1.340, train_macc=30.17, train_miou=11.29)]
Traceback (most recent call last):
  File "train.py", line 21, in <module>
    main()
  File "C:\Users\EAL\Anaconda3\envs\testenv6_git_inst\lib\site-packages\hydra\main.py", line 24, in decorated_main
    strict=strict,
  File "C:\Users\EAL\Anaconda3\envs\testenv6_git_inst\lib\site-packages\hydra\_internal\utils.py", line 174, in run_hydra
    overrides=args.overrides,
  File "C:\Users\EAL\Anaconda3\envs\testenv6_git_inst\lib\site-packages\hydra\_internal\hydra.py", line 86, in run
    job_subdir_key=None,
  File "C:\Users\EAL\Anaconda3\envs\testenv6_git_inst\lib\site-packages\hydra\plugins\common\utils.py", line 109, in run_job
    ret.return_value = task_function(task_cfg)
  File "train.py", line 13, in main
    trainer.train()
  File "C:\Users\EAL\Documents\prototyping\private_fork\torch-points3d-eurosense\torch_points3d\trainer.py", line 141, in train
    self._train_epoch(epoch)
  File "C:\Users\EAL\Documents\prototyping\private_fork\torch-points3d-eurosense\torch_points3d\trainer.py", line 195, in _train_epoch
    self._model.optimize_parameters(epoch, self._dataset.batch_size)
  File "C:\Users\EAL\Documents\prototyping\private_fork\torch-points3d-eurosense\torch_points3d\models\base_model.py", line 213, in optimize_parameters
    self.forward(epoch=epoch)  # first call forward to calculate intermediate results
  File "C:\Users\EAL\Documents\prototyping\private_fork\torch-points3d-eurosense\torch_points3d\models\segmentation\kpconv.py", line 115, in forward
    data = self.down_modules[-1](data, precomputed=self.pre_computed)
  File "C:\Users\EAL\Anaconda3\envs\testenv6_git_inst\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\EAL\Documents\prototyping\private_fork\torch-points3d-eurosense\torch_points3d\modules\KPConv\blocks.py", line 280, in forward
    data = block(data, precomputed=precomputed)
  File "C:\Users\EAL\Anaconda3\envs\testenv6_git_inst\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\EAL\Documents\prototyping\private_fork\torch-points3d-eurosense\torch_points3d\modules\KPConv\blocks.py", line 201, in forward
    output = self.kp_conv(output, precomputed=precomputed)
  File "C:\Users\EAL\Anaconda3\envs\testenv6_git_inst\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\EAL\Documents\prototyping\private_fork\torch-points3d-eurosense\torch_points3d\modules\KPConv\blocks.py", line 84, in forward
    idx_neighboors = self.neighbour_finder(data.pos, q_pos, batch_x=data.batch, batch_y=q_batch)
  File "C:\Users\EAL\Documents\prototyping\private_fork\torch-points3d-eurosense\torch_points3d\core\spatial_ops\neighbour_finder.py", line 15, in __call__
    return self.find_neighbours(x, y, batch_x, batch_y)
  File "C:\Users\EAL\Documents\prototyping\private_fork\torch-points3d-eurosense\torch_points3d\core\spatial_ops\neighbour_finder.py", line 36, in find_neighbours
    self._radius, self._max_num_neighbors, x, y, mode=self._conv_type, batch_x=batch_x, batch_y=batch_y
  File "C:\Users\EAL\Anaconda3\envs\testenv6_git_inst\lib\site-packages\torch_points_kernels\torchpoints.py", line 211, in ball_query
    return ball_query_partial_dense(radius, nsample, x, y, batch_x, batch_y, sort=sort)
  File "C:\Users\EAL\Anaconda3\envs\testenv6_git_inst\lib\site-packages\torch_points_kernels\torchpoints.py", line 168, in ball_query_partial_dense
    ind, dist = tpcpu.batch_ball_query(x, y, batch_x, batch_y, radius, nsample, mode=0, sorted=sort)
RuntimeError: Both batches need to have the same number of samples.

Just to be sure, I checked that the data loaded from the .pt file has consistent slices:

>>> data, slices = torch.load('train_set.pt')
>>>torch.equal(slices['x'],slices['y']))
True
>>>torch.equal(slices['x'],slices['pos'])
True
>>>torch.equal(slices['pos'],slices['y'])
True