Comments (5)
Hi, @filaPro . I think I have found the reason, it's because the GPU doesn't have enough memory. So I will close this issue. Thanks again for your help.
from fcaf3d.
Hi @week0425 ,
Can you please attach the complete .log
file and the command you are running?
from fcaf3d.
Command: python tools/train.py configs/fcaf3d/fcaf3d_s3dis-3d-5class.py
The error ocurred at the start of the first training session:
root@adc44dc27719:/mmdetection3d# python tools/train.py configs/fcaf3d/fcaf3d_s3dis-3d-5class.py
/opt/conda/lib/python3.8/site-packages/MinkowskiEngine/__init__.py:36: UserWarning: The environment variable `OMP_NUM_THREADS` not set. MinkowskiEngine will automatically set `OMP_NUM_THREADS=16`. If you want to set `OMP_NUM_THREADS` manually, please export it on the command line before running a python script. e.g. `export OMP_NUM_THREADS=12; python your_program.py`. It is recommended to set it below 24.
warnings.warn(
2022-07-26 06:44:37,904 - mmdet - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.8.8 (default, Feb 24 2021, 21:46:12) [GCC 7.3.0]
CUDA available: True
GPU 0: Quadro P5000
CUDA_HOME: /usr/local/cuda-11.1
NVCC: Build cuda_11.1.TC455_06.29190527_0
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.8.0
PyTorch compiling details: PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 11.1
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
- CuDNN 8.0.5
- Magma 2.5.2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.9.0
OpenCV: 4.5.5
MMCV: 1.3.9
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.1
MMDetection: 2.14.0
MMSegmentation: 0.14.1
MMDetection3D: 0.15.0+69d12eb
------------------------------------------------------------
2022-07-26 06:44:39,058 - mmdet - INFO - Distributed training: False
2022-07-26 06:44:40,266 - mmdet - INFO - Config:
voxel_size = 0.01
model = dict(
type='SingleStageSparse3DDetector',
voxel_size=0.01,
backbone=dict(type='MEResNet3D', in_channels=3, depth=34),
neck_with_head=dict(
type='Fcaf3DNeckWithHead',
in_channels=(64, 128, 256, 512),
out_channels=128,
pts_threshold=100000,
n_classes=5,
n_reg_outs=6,
voxel_size=0.01,
assigner=dict(type='Fcaf3DAssigner', limit=27, topk=18, n_scales=4),
loss_bbox=dict(type='IoU3DLoss', loss_weight=1.0, with_yaw=False)),
train_cfg=dict(),
test_cfg=dict(nms_pre=1000, iou_thr=0.5, score_thr=0.01))
optimizer = dict(type='AdamW', lr=0.001, weight_decay=0.0001)
optimizer_config = dict(grad_clip=dict(max_norm=10, norm_type=2))
lr_config = dict(policy='step', warmup=None, step=[8, 11])
runner = dict(type='EpochBasedRunner', max_epochs=50)
custom_hooks = [dict(type='EmptyCacheHook', after_iter=True)]
checkpoint_config = dict(interval=1, max_keep_ckpts=1)
log_config = dict(interval=50, hooks=[dict(type='TextLoggerHook')])
dist_params = dict(backend='nccl')
log_level = 'INFO'
work_dir = './work_dirs/fcaf3d_s3dis-3d-5class'
load_from = None
resume_from = None
workflow = [('train', 1)]
n_points = 100000
dataset_type = 'S3DISDataset'
data_root = './data/s3dis/'
class_names = ('table', 'chair', 'sofa', 'bookcase', 'board')
train_area = [1, 2, 3, 4, 6]
test_area = 5
train_pipeline = [
dict(
type='LoadPointsFromFile',
coord_type='DEPTH',
shift_height=False,
load_dim=6,
use_dim=[0, 1, 2, 3, 4, 5]),
dict(type='LoadAnnotations3D'),
dict(type='IndoorPointSample', num_points=100000),
dict(
type='RandomFlip3D',
sync_2d=False,
flip_ratio_bev_horizontal=0.5,
flip_ratio_bev_vertical=0.5),
dict(
type='GlobalRotScaleTrans',
rot_range=[-0.087266, 0.087266],
scale_ratio_range=[0.9, 1.1],
translation_std=[0.1, 0.1, 0.1],
shift_height=False),
dict(
type='DefaultFormatBundle3D',
class_names=('table', 'chair', 'sofa', 'bookcase', 'board')),
dict(type='Collect3D', keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
]
test_pipeline = [
dict(
type='LoadPointsFromFile',
coord_type='DEPTH',
shift_height=False,
load_dim=6,
use_dim=[0, 1, 2, 3, 4, 5]),
dict(
type='MultiScaleFlipAug3D',
img_scale=(1333, 800),
pts_scale_ratio=1,
flip=False,
transforms=[
dict(
type='GlobalRotScaleTrans',
rot_range=[0, 0],
scale_ratio_range=[1.0, 1.0],
translation_std=[0, 0, 0]),
dict(
type='RandomFlip3D',
sync_2d=False,
flip_ratio_bev_horizontal=0.5,
flip_ratio_bev_vertical=0.5),
dict(type='IndoorPointSample', num_points=100000),
dict(
type='DefaultFormatBundle3D',
class_names=('table', 'chair', 'sofa', 'bookcase', 'board'),
with_label=False),
dict(type='Collect3D', keys=['points'])
])
]
data = dict(
samples_per_gpu=8,
workers_per_gpu=4,
train=dict(
type='RepeatDataset',
times=13,
dataset=dict(
type='ConcatDataset',
datasets=[
dict(
type='S3DISDataset',
data_root='./data/s3dis/',
ann_file='./data/s3dis/s3dis_infos_Area_1.pkl',
pipeline=[
dict(
type='LoadPointsFromFile',
coord_type='DEPTH',
shift_height=False,
load_dim=6,
use_dim=[0, 1, 2, 3, 4, 5]),
dict(type='LoadAnnotations3D'),
dict(type='IndoorPointSample', num_points=100000),
dict(
type='RandomFlip3D',
sync_2d=False,
flip_ratio_bev_horizontal=0.5,
flip_ratio_bev_vertical=0.5),
dict(
type='GlobalRotScaleTrans',
rot_range=[-0.087266, 0.087266],
scale_ratio_range=[0.9, 1.1],
translation_std=[0.1, 0.1, 0.1],
shift_height=False),
dict(
type='DefaultFormatBundle3D',
class_names=('table', 'chair', 'sofa', 'bookcase',
'board')),
dict(
type='Collect3D',
keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
],
filter_empty_gt=True,
classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
box_type_3d='Depth'),
dict(
type='S3DISDataset',
data_root='./data/s3dis/',
ann_file='./data/s3dis/s3dis_infos_Area_2.pkl',
pipeline=[
dict(
type='LoadPointsFromFile',
coord_type='DEPTH',
shift_height=False,
load_dim=6,
use_dim=[0, 1, 2, 3, 4, 5]),
dict(type='LoadAnnotations3D'),
dict(type='IndoorPointSample', num_points=100000),
dict(
type='RandomFlip3D',
sync_2d=False,
flip_ratio_bev_horizontal=0.5,
flip_ratio_bev_vertical=0.5),
dict(
type='GlobalRotScaleTrans',
rot_range=[-0.087266, 0.087266],
scale_ratio_range=[0.9, 1.1],
translation_std=[0.1, 0.1, 0.1],
shift_height=False),
dict(
type='DefaultFormatBundle3D',
class_names=('table', 'chair', 'sofa', 'bookcase',
'board')),
dict(
type='Collect3D',
keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
],
filter_empty_gt=True,
classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
box_type_3d='Depth'),
dict(
type='S3DISDataset',
data_root='./data/s3dis/',
ann_file='./data/s3dis/s3dis_infos_Area_3.pkl',
pipeline=[
dict(
type='LoadPointsFromFile',
coord_type='DEPTH',
shift_height=False,
load_dim=6,
use_dim=[0, 1, 2, 3, 4, 5]),
dict(type='LoadAnnotations3D'),
dict(type='IndoorPointSample', num_points=100000),
dict(
type='RandomFlip3D',
sync_2d=False,
flip_ratio_bev_horizontal=0.5,
flip_ratio_bev_vertical=0.5),
dict(
type='GlobalRotScaleTrans',
rot_range=[-0.087266, 0.087266],
scale_ratio_range=[0.9, 1.1],
translation_std=[0.1, 0.1, 0.1],
shift_height=False),
dict(
type='DefaultFormatBundle3D',
class_names=('table', 'chair', 'sofa', 'bookcase',
'board')),
dict(
type='Collect3D',
keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
],
filter_empty_gt=True,
classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
box_type_3d='Depth'),
dict(
type='S3DISDataset',
data_root='./data/s3dis/',
ann_file='./data/s3dis/s3dis_infos_Area_4.pkl',
pipeline=[
dict(
type='LoadPointsFromFile',
coord_type='DEPTH',
shift_height=False,
load_dim=6,
use_dim=[0, 1, 2, 3, 4, 5]),
dict(type='LoadAnnotations3D'),
dict(type='IndoorPointSample', num_points=100000),
dict(
type='RandomFlip3D',
sync_2d=False,
flip_ratio_bev_horizontal=0.5,
flip_ratio_bev_vertical=0.5),
dict(
type='GlobalRotScaleTrans',
rot_range=[-0.087266, 0.087266],
scale_ratio_range=[0.9, 1.1],
translation_std=[0.1, 0.1, 0.1],
shift_height=False),
dict(
type='DefaultFormatBundle3D',
class_names=('table', 'chair', 'sofa', 'bookcase',
'board')),
dict(
type='Collect3D',
keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
],
filter_empty_gt=True,
classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
box_type_3d='Depth'),
dict(
type='S3DISDataset',
data_root='./data/s3dis/',
ann_file='./data/s3dis/s3dis_infos_Area_6.pkl',
pipeline=[
dict(
type='LoadPointsFromFile',
coord_type='DEPTH',
shift_height=False,
load_dim=6,
use_dim=[0, 1, 2, 3, 4, 5]),
dict(type='LoadAnnotations3D'),
dict(type='IndoorPointSample', num_points=100000),
dict(
type='RandomFlip3D',
sync_2d=False,
flip_ratio_bev_horizontal=0.5,
flip_ratio_bev_vertical=0.5),
dict(
type='GlobalRotScaleTrans',
rot_range=[-0.087266, 0.087266],
scale_ratio_range=[0.9, 1.1],
translation_std=[0.1, 0.1, 0.1],
shift_height=False),
dict(
type='DefaultFormatBundle3D',
class_names=('table', 'chair', 'sofa', 'bookcase',
'board')),
dict(
type='Collect3D',
keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
],
filter_empty_gt=True,
classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
box_type_3d='Depth')
],
separate_eval=False)),
val=dict(
type='S3DISDataset',
data_root='./data/s3dis/',
ann_file='./data/s3dis/s3dis_infos_Area_5.pkl',
pipeline=[
dict(
type='LoadPointsFromFile',
coord_type='DEPTH',
shift_height=False,
load_dim=6,
use_dim=[0, 1, 2, 3, 4, 5]),
dict(
type='MultiScaleFlipAug3D',
img_scale=(1333, 800),
pts_scale_ratio=1,
flip=False,
transforms=[
dict(
type='GlobalRotScaleTrans',
rot_range=[0, 0],
scale_ratio_range=[1.0, 1.0],
translation_std=[0, 0, 0]),
dict(
type='RandomFlip3D',
sync_2d=False,
flip_ratio_bev_horizontal=0.5,
flip_ratio_bev_vertical=0.5),
dict(type='IndoorPointSample', num_points=100000),
dict(
type='DefaultFormatBundle3D',
class_names=('table', 'chair', 'sofa', 'bookcase',
'board'),
with_label=False),
dict(type='Collect3D', keys=['points'])
])
],
classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
test_mode=True,
box_type_3d='Depth'),
test=dict(
type='S3DISDataset',
data_root='./data/s3dis/',
ann_file='./data/s3dis/s3dis_infos_Area_5.pkl',
pipeline=[
dict(
type='LoadPointsFromFile',
coord_type='DEPTH',
shift_height=False,
load_dim=6,
use_dim=[0, 1, 2, 3, 4, 5]),
dict(
type='MultiScaleFlipAug3D',
img_scale=(1333, 800),
pts_scale_ratio=1,
flip=False,
transforms=[
dict(
type='GlobalRotScaleTrans',
rot_range=[0, 0],
scale_ratio_range=[1.0, 1.0],
translation_std=[0, 0, 0]),
dict(
type='RandomFlip3D',
sync_2d=False,
flip_ratio_bev_horizontal=0.5,
flip_ratio_bev_vertical=0.5),
dict(type='IndoorPointSample', num_points=100000),
dict(
type='DefaultFormatBundle3D',
class_names=('table', 'chair', 'sofa', 'bookcase',
'board'),
with_label=False),
dict(type='Collect3D', keys=['points'])
])
],
classes=('table', 'chair', 'sofa', 'bookcase', 'board'),
test_mode=True,
box_type_3d='Depth'))
gpu_ids = range(0, 1)
2022-07-26 06:44:40,267 - mmdet - INFO - Set random seed to 0, deterministic: False
2022-07-26 06:44:41,884 - mmdet - INFO - Model:
SingleStageSparse3DDetector(
(backbone): MEResNet3D(
(conv1): Sequential(
(0): MinkowskiConvolution(in=3, out=64, kernel_size=[3, 3, 3], stride=[2, 2, 2], dilation=[1, 1, 1])
(1): MinkowskiInstanceNorm(nchannels=64)
(2): MinkowskiReLU()
(3): MinkowskiMaxPooling(kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1])
)
(layer1): Sequential(
(0): BasicBlock(
(conv1): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[2, 2, 2], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
(downsample): Sequential(
(0): MinkowskiConvolution(in=64, out=64, kernel_size=[1, 1, 1], stride=[2, 2, 2], dilation=[1, 1, 1])
(1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
)
(2): BasicBlock(
(conv1): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
)
)
(layer2): Sequential(
(0): BasicBlock(
(conv1): MinkowskiConvolution(in=64, out=128, kernel_size=[3, 3, 3], stride=[2, 2, 2], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
(downsample): Sequential(
(0): MinkowskiConvolution(in=64, out=128, kernel_size=[1, 1, 1], stride=[2, 2, 2], dilation=[1, 1, 1])
(1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
)
(2): BasicBlock(
(conv1): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
)
(3): BasicBlock(
(conv1): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
)
)
(layer3): Sequential(
(0): BasicBlock(
(conv1): MinkowskiConvolution(in=128, out=256, kernel_size=[3, 3, 3], stride=[2, 2, 2], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
(downsample): Sequential(
(0): MinkowskiConvolution(in=128, out=256, kernel_size=[1, 1, 1], stride=[2, 2, 2], dilation=[1, 1, 1])
(1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
)
(2): BasicBlock(
(conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
)
(3): BasicBlock(
(conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
)
(4): BasicBlock(
(conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
)
(5): BasicBlock(
(conv1): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
)
)
(layer4): Sequential(
(0): BasicBlock(
(conv1): MinkowskiConvolution(in=256, out=512, kernel_size=[3, 3, 3], stride=[2, 2, 2], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=512, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
(downsample): Sequential(
(0): MinkowskiConvolution(in=256, out=512, kernel_size=[1, 1, 1], stride=[2, 2, 2], dilation=[1, 1, 1])
(1): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): MinkowskiConvolution(in=512, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=512, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
)
(2): BasicBlock(
(conv1): MinkowskiConvolution(in=512, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm1): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(conv2): MinkowskiConvolution(in=512, out=512, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(norm2): MinkowskiBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): MinkowskiReLU()
)
)
)
(neck_with_head): Fcaf3DNeckWithHead(
(loss_centerness): CrossEntropyLoss()
(loss_bbox): IoU3DLoss()
(loss_cls): FocalLoss()
(pruning): MinkowskiPruning()
(out_block_0): Sequential(
(0): MinkowskiConvolution(in=64, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): MinkowskiELU()
)
(up_block_1): Sequential(
(0): MinkowskiGenerativeConvolutionTranspose(in=128, out=64, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1])
(1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): MinkowskiELU()
(3): MinkowskiConvolution(in=64, out=64, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(4): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): MinkowskiELU()
)
(out_block_1): Sequential(
(0): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): MinkowskiELU()
)
(up_block_2): Sequential(
(0): MinkowskiGenerativeConvolutionTranspose(in=256, out=128, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1])
(1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): MinkowskiELU()
(3): MinkowskiConvolution(in=128, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(4): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): MinkowskiELU()
)
(out_block_2): Sequential(
(0): MinkowskiConvolution(in=256, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): MinkowskiELU()
)
(up_block_3): Sequential(
(0): MinkowskiGenerativeConvolutionTranspose(in=512, out=256, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1])
(1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): MinkowskiELU()
(3): MinkowskiConvolution(in=256, out=256, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(4): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): MinkowskiELU()
)
(out_block_3): Sequential(
(0): MinkowskiConvolution(in=512, out=128, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1])
(1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): MinkowskiELU()
)
(centerness_conv): MinkowskiConvolution(in=128, out=1, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1])
(reg_conv): MinkowskiConvolution(in=128, out=6, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1])
(cls_conv): MinkowskiConvolution(in=128, out=5, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1])
(scales): ModuleList(
(0): Scale()
(1): Scale()
(2): Scale()
(3): Scale()
)
)
)
./data/s3dis/s3dis_infos_Area_1.pkl
./data/s3dis/s3dis_infos_Area_2.pkl
./data/s3dis/s3dis_infos_Area_3.pkl
./data/s3dis/s3dis_infos_Area_4.pkl
./data/s3dis/s3dis_infos_Area_6.pkl
./data/s3dis/s3dis_infos_Area_5.pkl
2022-07-26 06:44:45,311 - mmdet - INFO - Start running, host: root@adc44dc27719, work_dir: /mmdetection3d/work_dirs/fcaf3d_s3dis-3d-5class
2022-07-26 06:44:45,311 - mmdet - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH ) StepLrUpdaterHook
(NORMAL ) CheckpointHook
(NORMAL ) EvalHook
(VERY_LOW ) TextLoggerHook
--------------------
before_train_epoch:
(VERY_HIGH ) StepLrUpdaterHook
(NORMAL ) EvalHook
(NORMAL ) EmptyCacheHook
(LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook
--------------------
before_train_iter:
(VERY_HIGH ) StepLrUpdaterHook
(NORMAL ) EvalHook
(LOW ) IterTimerHook
--------------------
after_train_iter:
(ABOVE_NORMAL) OptimizerHook
(NORMAL ) CheckpointHook
(NORMAL ) EvalHook
(NORMAL ) EmptyCacheHook
(LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook
--------------------
after_train_epoch:
(NORMAL ) CheckpointHook
(NORMAL ) EvalHook
(NORMAL ) EmptyCacheHook
(VERY_LOW ) TextLoggerHook
--------------------
before_val_epoch:
(NORMAL ) EmptyCacheHook
(LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook
--------------------
before_val_iter:
(LOW ) IterTimerHook
--------------------
after_val_iter:
(NORMAL ) EmptyCacheHook
(LOW ) IterTimerHook
--------------------
after_val_epoch:
(NORMAL ) EmptyCacheHook
(VERY_LOW ) TextLoggerHook
--------------------
2022-07-26 06:44:45,311 - mmdet - INFO - workflow: [('train', 1)], max: 50 epochs
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [1,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [2,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [3,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [4,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [5,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [6,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [7,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [8,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [9,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [10,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [11,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [12,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [13,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [14,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [15,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [16,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [17,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [18,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [19,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [20,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [21,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [22,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [23,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [24,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [25,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [26,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [27,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [28,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [29,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [30,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [31,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [96,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [98,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [99,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [100,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [101,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [102,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [103,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [104,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [105,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [106,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [107,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [108,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [109,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [110,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [111,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [112,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [113,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [114,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [115,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [116,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [117,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [118,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [119,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [120,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [121,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [122,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [123,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [124,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [125,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [126,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [127,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [96,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [98,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [99,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [100,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [101,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [102,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [103,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [104,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [105,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [106,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [107,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [108,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [109,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [110,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [111,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [112,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [113,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [114,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [115,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [116,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [117,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [118,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [119,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [120,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [121,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [122,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [123,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [124,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [125,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [126,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [127,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [64,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [65,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [66,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [67,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [68,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [69,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [70,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [71,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [72,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [73,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [74,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [75,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [76,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [77,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [78,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [79,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [80,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [81,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [82,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [83,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [84,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [85,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [86,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [87,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [88,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [89,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [90,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [91,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [92,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [93,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [94,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [60,0,0], thread: [95,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
Traceback (most recent call last):
File "tools/train.py", line 223, in <module>
main()
File "tools/train.py", line 212, in main
train_model(
File "/mmdetection3d/mmdet3d/apis/train.py", line 27, in train_model
train_detector(
File "/opt/conda/lib/python3.8/site-packages/mmdet/apis/train.py", line 170, in train_detector
runner.run(data_loaders, cfg.workflow)
File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run
epoch_runner(data_loaders[i], **kwargs)
File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train
self.run_iter(data_batch, train_mode=True, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 29, in run_iter
outputs = self.model.train_step(data_batch, self.optimizer,
File "/opt/conda/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 67, in train_step
return self.module.train_step(*inputs[0], **kwargs[0])
File "/opt/conda/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 237, in train_step
losses = self(**data)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 98, in new_func
return old_func(*args, **kwargs)
File "/mmdetection3d/mmdet3d/models/detectors/base.py", line 58, in forward
return self.forward_train(**kwargs)
File "/mmdetection3d/mmdet3d/models/detectors/single_stage_sparse.py", line 48, in forward_train
x = self.extract_feat(points, img_metas)
File "/mmdetection3d/mmdet3d/models/detectors/single_stage_sparse.py", line 40, in extract_feat
x = self.neck_with_head(x)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/mmdetection3d/mmdet3d/models/dense_heads/fcaf3d_neck_with_head.py", line 102, in forward
x = self._prune(x, scores)
File "/mmdetection3d/mmdet3d/models/dense_heads/fcaf3d_neck_with_head.py", line 124, in _prune
prune_mask[permutation[mask]] = True
RuntimeError: CUDA error: device-side assert triggered
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered
Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1614378083779/work/c10/cuda/CUDACachingAllocator.cpp:733 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fb5e39412f2 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x5b (0x7fb5e393e67b in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x809 (0x7fb5e3b9a219 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x54 (0x7fb5e39293a4 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #4: <unknown function> + 0x6e0dda (0x7fb63a89ddda in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x6e0e71 (0x7fb63a89de71 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #26: __libc_start_main + 0xe7 (0x7fb675ef4bf7 in /lib/x86_64-linux-gnu/libc.so.6)
from fcaf3d.
Can you please try with our dockerfile or package versions from there, i.e. cuda 10.2
? I also didn't try to train on a single gpu, only on 2 gpus. And one more thing, you will run out of memory for S3DIS dataset with your 16 Gb gpu, or you should decrease batch size. Btw, can you successfully run test.py
on S3DIS with our pretrained model?
However anyway the error looks strange. I can have a look if nothing from above helps you...
from fcaf3d.
OK, I will try. I have tried to train S3DIS on the pointnet2. It worked well. So I think it may be not the environment's problem. I will give you a feedback when I find new messages. Thank you!
from fcaf3d.
Related Issues (20)
- RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1603729009598/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, invalid usage, NCCL version 2.7.8 HOT 1
- ModuleNotFoundError: No module named 'mmdet3d.ops.pcdet_nms'
- visualization question HOT 7
- train.py: error: unrecognized arguments: --load-from fully_conv_s3dis.pth HOT 1
- Can MinkowskiGenerativeConvolutionTranspose from fcaf3d_neck_with_head.py replaced? HOT 4
- output all predict bbox coordinate position HOT 4
- log information HOT 1
- run pcd_demo.py HOT 3
- mAP result question HOT 2
- Performance on ScanNet HOT 4
- resume-from
- Performance on SUN RGB-D HOT 5
- Concatenate different feature levels to one level before passing data to the heads HOT 2
- terminate called after throwing an instance of 'c10::CUDAError' HOT 1
- Generalization of FCAF3D HOT 2
- FCAF3D evaluation result mAP question HOT 2
- Training with Custom Dataset HOT 1
- Welcome update to OpenMMLab 2.0 HOT 1
- How to solve the error reported during the training trial HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fcaf3d.