Segmentation fault of benchdnn with Intel oneAPI/NVIDIA plugin

[vitduck]

Segmentation fault of benchdnn with Intel oneAPI/NVIDIA plugin

[mehdi-goli]

@sgeor255 , @dylan-angus-codeplay could you please look into that please.

[vitduck]

Hi @sgeor255,

Thanks for trying to reproduce the issue on your end.
We could not install version 2023.2.1 since it is not available to download.

For now, we will conduct tests with an explicit ONEAPI_DEVICE_SELECTOR.
Once 2023.2.1 is available to download, we will test again and update the thread.

Thanks very much for your time.

[vpirogov]

+@mehdi-goli

[vitduck]

Hi @vpirogov

The debug with SYCL_PI_TRACE=1 shows that the correct CUDA backend is selected

$  SYCL_PI_TRACE=1 ./benchdnn --engine=gpu:0 --binary 3x4x5:3x4x5 
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_opencl.so [ PluginVersion: 12.27.1 ]
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_cuda.so [ PluginVersion: 12.27.1 ]
SYCL_PI_TRACE[all]: Requested device_type: info::device_type::gpu
SYCL_PI_TRACE[all]: Requested device_type: info::device_type::gpu
SYCL_PI_TRACE[all]: Requested device_type: info::device_type::gpu
SYCL_PI_TRACE[all]: Selected device: -> final score = 2000
SYCL_PI_TRACE[all]:   platform: NVIDIA CUDA BACKEND
SYCL_PI_TRACE[all]:   device: NVIDIA A100-SXM4-80GB
Segmentation fault (core dumped)

I then realized that the cpu backend needs to be explicitly specified as well

$  ONEAPI_DEVICE_SELECTOR="cuda:*" ./benchdnn --engine=gpu:0 --binary 3x4x5:3x4x5
ERROR: can't create CPU engine. Possible reasons for this error:
- Incorrect SYCL_DEVICE_FILTER. The filter must be either unset or include 'opencl:cpu' devices.
- Missing TBB library which is required for OpenCL CPU runtime. Check that TBB library is available in the system.
- Missing OpenCL CPU runtime or other issues with OpenCL CPU runtime. Check that output from `sycl-ls` or `clinfo -l` commands include any CPU devices.
error [engine_t::engine_t(dnnl_engine_kind_t):1215]: 'status' -> invalid_arguments(2)

Per the first suggestion, the correct way to run benchdnn would be

$ ONEAPI_DEVICE_SELECTOR="opencl:cpu;cuda:*" ./benchdnn --engine=gpu:0 --binary 3x4x5:3x4x5
0:PASSED __REPRO: --binary --engine=gpu 3x4x5:3x4x5
tests:1 passed:1 skipped:0 mistrusted:0 unimplemented:0 invalid_arguments:0 failed:0 listed:0
total compute_ref: sum(s):0.00

Though I am somewhat confused regarding the default behavior when ONEAPI_DEVICE_SELECTOR is not set.
The debug message shows that libpi_opencl.so is found and successfully loaded.
Therefore , there should be no issue with the creation of CPU engine.

[vpirogov]

Thanks for update, @vitduck. We definitely should not segfault with default ONEAPI_DEVICE_SELECTOR setting.

[sgeor255]

Thanks @vpirogov , we are looking into this.

[sgeor255]

Hi, we tried to reproduce the segfault reported above, however with our setup (oneAPI basekit 2023.2.1, cuda 11.7.1, cudnn 8.9.0), we can't reproduce the segfault. From the follow up comment it looks like @vitduck has managed to work around the issue, however if this is still a problem we would suggest updating the dependencies (most likely the oneAPI basekit version is the culprit) and rerunning the test. If you the segfault is still seen, please let us know and we will look into it further. Thanks

[vpirogov]

Confirmed that the issue does not reproduce with oneAPI 2024.0 and oneDNN v3.3.3:

$ DNNL_VERBOSE=1 ./benchdnn --engine=gpu --binary 3x4x5:3x4x5
onednn_verbose,info,oneDNN v3.3.3 (commit c0ae38cdf1201caf8ffd2906077defdfe7f4aaa3)
onednn_verbose,info,cpu,runtime:DPC++,nthr:48
onednn_verbose,info,cpu,isa:Intel AVX-512 with Intel DL Boost
onednn_verbose,info,gpu,runtime:DPC++
onednn_verbose,info,cpu,engine,0,backend:OpenCL,name:Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz,driver_version:2023.16.10,binary_kernels:disabled
onednn_verbose,info,gpu,engine,0,backend:Nvidia,name:NVIDIA A10,driver_version:0.0.0,binary_kernels:disabled
onednn_verbose,primitive,info,template:operation,engine,primitive,implementation,prop_kind,memory_descriptors,attributes,auxiliary,problem_desc,exec_time
onednn_verbose,primitive,exec,cpu,reorder,jit:uni,undef,src_f32::blocked:abc::f0 dst_f32::blocked:abc::f0,,,3x4x5,0.440186
onednn_verbose,primitive,exec,cpu,reorder,jit:uni,undef,src_f32::blocked:abc::f0 dst_f32::blocked:abc::f0,,,3x4x5,0.24292
onednn_verbose,primitive,exec,gpu,binary,cuda:cudnn:any,undef,src_f32::blocked:abc::f0 src_f32:a:blocked:abc::f0 dst_f32:a:blocked:abc::f0,,alg:binary_add,3x4x5:3x4x5,1.53296
onednn_verbose,primitive,exec,cpu,reorder,jit:uni,undef,src_f32::blocked:abc::f0 dst_f32::blocked:abc::f0,,,3x4x5,0.295898
0:PASSED __REPRO: --binary --engine=gpu 3x4x5:3x4x5
tests:1 passed:1 skipped:0 mistrusted:0 unimplemented:0 invalid_arguments:0 failed:0 listed:0
total: 2.04s; fill: 0.09s (4%); compute_ref: 0.00s (0%); compare: 0.00s (0%);