rocmarchive/realcaffe2

The repo is obsolete. Use at your own risk.

realcaffe2's Issues

Verify Lenet, Cifar10 on sigle node env

Fix group_ implementation in Conv op

Seeing miopen error when running AlexNet. This issue is not caught by the bundle unit test.

Verify AlexNet, VGG, Resnet

Exposing MIOPEN to python scripts

Is there a way to expose availability of MIOPEN to python scripts? Because, in places where CUDNN is specified as engine for the operators, it doesn't seem appropriate to check "has_hip" condition to specify MIOPEN as engine, since it misses the cases where hip is not present but MIOPEN is present. @petrex @ashishfarmer

caffe2 utility binary loading shared libraries libcaffe2_hip.so

@ashishfarmer
For hip binary tests we need libcaffe2_hip.so.
How about other utility binaries?Is it necessary? pls share your insight. thx.

pyeh@rocm-miopen /usr/local/bin $ ./make_cifar_db                                                                                                               
./make_cifar_db: error while loading shared libraries: ../../lib/libcaffe2_hip.so: cannot open shared object file: No such file or directory

Validation on multi GPU env

Requires RCCL.

Investigate and fix compiler warning -- warning: the use of `tmpnam' is dangerous, better use `mkstemp'

CMakeFiles/text_file_reader_utils_test.dir/operators/text_file_reader_utils_test.cc.o: In function caffe2::TextFileReaderUtilsTest_TokenizeTest_Test::TestBody()': text_file_reader_utils_test.cc:(.text+0x1f78): warning: the use of tmpnam' is dangerous, better use `mkstemp'

Operator tests with cub/thrust dependency

Create a list of cub dependent tests. (deferred for now)
We would need to revisit it when RocPRIM is ready.

• caffe2/operators/elementwise_op_hip.cc
• caffe2/operators/accuracy_op_hip.cc

Investigate and fix compiler warning from operator.h [-Winconsistent-missing-override]

/home/pyeh/Projects/rocm_caffe2/caffe2/core/operator.h:347:8: warning: 'WaitEvent' overrides a member function but is not marked 'override' [-Winconsistent-missing-override]
void WaitEvent(const Event& ev, int stream_id = -1) final {

mipoen integration

HIP FP32 intrinsics

Identify HIP equivalent for FP32 intrinsics __fmaf_rn, __fmul_rn, __fsub_rn and fix the HIP ops

Engage CQE team for Validation

Verify RNN

add official dockerfile for AMD platform.

Currently, the ops are validated on rocm 1.7.1.rc5

Validate hipified operators with extra dependencies (thrust/cub/eigen... etc)

BatchMatMul is failing

Errors for float16 type.
GPU error

ERROR: test_batch_matmul (main.TestBatchMatMul)

Traceback (most recent call last):
File "../caffe2/python/operator_test/matmul_op_test.py", line 145, in test_batch_matmul
@given(
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 1049, in wrapped_test
state.run()
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 820, in run
falsifying_example.__expected_traceback,
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 581, in execute
result = self.test_runner(data, run)
File "/usr/local/lib/python2.7/dist-packages/hypothesis/executors.py", line 58, in default_new_style_executor
return function(data)
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 573, in run
return test(*args, **kwargs)
File "../caffe2/python/operator_test/matmul_op_test.py", line 145, in test_batch_matmul
@given(
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 520, in test
result = self.test(*args, **kwargs)
File "../caffe2/python/operator_test/matmul_op_test.py", line 195, in test_batch_matmul
relax_fp16_check(self.assertReferenceChecks, gc, op, [X, Y, trans_a, trans_b, dtype], matmul_ref)
File "../caffe2/python/operator_test/matmul_op_test.py", line 192, in relax_fp16_check
check_func(*args, threshold=threshold, **kwargs)
File "/data/rocm_caffe2/build/caffe2/python/hypothesis_test_util.py", line 574, in assertReferenceChecks
workspace.RunNetOnce(net)
File "/data/rocm_caffe2/build/caffe2/python/workspace.py", line 216, in RunNetOnce
StringifyProto(net),
File "/data/rocm_caffe2/build/caffe2/python/workspace.py", line 199, in CallWithExceptionIntercept
return func(*args, **kwargs)
RuntimeError: [enforce fail at math_hip.cc:400] . Unsupported math type Error from operator:
input: "X" input: "Y" output: "out" name: "" type: "BatchMatMul" arg { name: "trans_a" i: 0 } arg { name: "trans_b" i: 0 } device_option { device_type: 4 }

CPU error

ERROR: test_batch_matmul (main.TestBatchMatMul)

Traceback (most recent call last):
File "../caffe2/python/operator_test/matmul_op_test.py", line 145, in test_batch_matmul
@given(
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 1049, in wrapped_test
state.run()
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 820, in run
falsifying_example.__expected_traceback,
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 581, in execute
result = self.test_runner(data, run)
File "/usr/local/lib/python2.7/dist-packages/hypothesis/executors.py", line 58, in default_new_style_executor
return function(data)
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 573, in run
return test(*args, **kwargs)
File "../caffe2/python/operator_test/matmul_op_test.py", line 145, in test_batch_matmul
@given(
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 520, in test
result = self.test(*args, **kwargs)
File "../caffe2/python/operator_test/matmul_op_test.py", line 195, in test_batch_matmul
relax_fp16_check(self.assertReferenceChecks, gc, op, [X, Y, trans_a, trans_b, dtype], matmul_ref)
File "../caffe2/python/operator_test/matmul_op_test.py", line 192, in relax_fp16_check
check_func(*args, threshold=threshold, **kwargs)
File "/data/rocm_caffe2/build/caffe2/python/hypothesis_test_util.py", line 574, in assertReferenceChecks
workspace.RunNetOnce(net)
File "/data/rocm_caffe2/build/caffe2/python/workspace.py", line 216, in RunNetOnce
StringifyProto(net),
File "/data/rocm_caffe2/build/caffe2/python/workspace.py", line 199, in CallWithExceptionIntercept
return func(*args, **kwargs)
RuntimeError: [enforce fail at operator.h:640] . Unsupported type of tensor: caffe2::__f16 Error from operator:
input: "X" input: "Y" output: "out" name: "" type: "BatchMatMul" arg { name: "trans_a" i: 0 } arg { name: "trans_b" i: 0 } device_option { }

For float32:
CPU is passing but gpu is hanging for gradient check..

pooling test fail when using MIOPEN as engine

miopen poolinf test fail.

Trying example: test_global_avg_pool_nchw(self=<__main__.TestPooling testMethod=test_global_avg_pool_nchw>, op_type='AveragePool2D', sz=2, batch_size=2, engine='MIOPEN', gc=device_type: 4, dc=[, device_type: 4])
*** Aborted at 1521584246 (unix time) try "date -d @1521584246" if you are using GNU date ***
PC: @     0x7fdf717181b1 miopen::PoolingDescriptor::GetForwardOutputDim()
*** SIGSEGV (@0x0) received by PID 62799 (TID 0x7fdfa3e49700) from PID 0; stack trace: ***
    @     0x7fdfa3a39390 (unknown)
    @     0x7fdf717181b1 miopen::PoolingDescriptor::GetForwardOutputDim()
    @     0x7fdf7158e838 miopenGetPoolingForwardOutputDim
    @     0x7fdf95e7e807 caffe2::MIOPENPoolOp::DoRunWithType<>()
    @     0x7fdf95e71f56 caffe2::MIOPENPoolOp::RunOnDevice()
    @     0x7fdf94a661fe caffe2::Operator<>::Run()
    @     0x7fdf76cf6732 caffe2::Workspace::RunOperatorOnce()
    @     0x7fdf9754ab4a _ZZN6caffe26python16addGlobalMethodsERN8pybind116moduleEENKUlRKNS1_5bytesEE26_clES6_.isra.2495.constprop.2527
    @     0x7fdf9754ade5 _ZZN8pybind1112cpp_function10initializeIZN6caffe26python16addGlobalMethodsERNS_6moduleEEUlRKNS_5bytesEE26_bJS8_EJNS_4nameENS_5scopeENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNESQ_
    @     0x7fdf9756ad43 pybind11::cpp_function::dispatcher()
    @           0x4c1d50 PyEval_EvalFrameEx
    @           0x4b9ab6 PyEval_EvalCodeEx
    @           0x4c1e6f PyEval_EvalFrameEx
    @           0x4b9ab6 PyEval_EvalCodeEx
    @           0x4c16e7 PyEval_EvalFrameEx
    @           0x4b9ab6 PyEval_EvalCodeEx
    @           0x4c16e7 PyEval_EvalFrameEx
    @           0x4b9ab6 PyEval_EvalCodeEx
    @           0x4d55f3 (unknown)
    @           0x4a577e PyObject_Call
    @           0x4bed3d PyEval_EvalFrameEx
    @           0x4b9ab6 PyEval_EvalCodeEx
    @           0x4c1e6f PyEval_EvalFrameEx
    @           0x4b9ab6 PyEval_EvalCodeEx
    @           0x4d55f3 (unknown)
    @           0x4a577e PyObject_Call
    @           0x4bed3d PyEval_EvalFrameEx
    @           0x4b9ab6 PyEval_EvalCodeEx
    @           0x4c1e6f PyEval_EvalFrameEx
    @           0x4b9ab6 PyEval_EvalCodeEx
    @           0x4c1e6f PyEval_EvalFrameEx
    @           0x4b9ab6 PyEval_EvalCodeEx
^C[1]    62799 segmentation fault (core dumped)  python -m caffe2.python.operator_test.pooling_test

Investigate and fix compile warning from clang-6.0 [-Wunused-command-line-argument]

clang-6.0: warning: /home/pyeh/Projects/rocm_caffe2/build/confu-srcs/pthreadpool/include: 'linker' input unused [-Wunused-command-line-argument]

Performace Tuning

add hip device support in hypothesis_test_util.py

We need to add device option for hip.

ln 268 of

hypothesis_test_util.py

@ashishfarmer I remember you had some prior work on this. any update? thanks

Add HIP device support in the python based operator tests

We need to add hip device support int the following 3 op test.

==================================== ERRORS ====================================
_____________ ERROR collecting python/data_parallel_model_test.py ______________
python/data_parallel_model_test.py:516: in
@unittest.skipIf(workspace.NumCudaDevices() < 2, "Need at least 2 GPUs.")
E AttributeError: 'module' object has no attribute 'NumCudaDevices'
------------------------------- Captured stdout --------------------------------
Ignoring @/caffe2/caffe2/contrib/nccl:nccl_ops as it is not a valid file.
Ignoring @/caffe2/caffe2/contrib/gloo:gloo_ops as it is not a valid file.
Ignoring @/caffe2/caffe2/contrib/gloo:gloo_ops_gpu as it is not a valid file.
Ignoring @/caffe2/caffe2/distributed:file_store_handler_ops as it is not a valid file.
________________ ERROR collecting python/gradient_check_test.py ________________
python/gradient_check_test.py:41: in
if workspace.has_gpu_support and workspace.NumCudaDevices() > 0:
E AttributeError: 'module' object has no attribute 'NumCudaDevices'
___________ ERROR collecting python/operator_test/load_save_test.py ____________
python/operator_test/load_save_test.py:34: in
max_gpuid = workspace.NumCudaDevices() - 1
E AttributeError: 'module' object has no attribute 'NumCudaDevices'

enable CI for rocm_caffe2 proejct

share the workspace between fwd/bwd pass for pooling op

Fix the HIP assert issues

Identify the assert functionality and in HIP and the roadmap. Fix the HIP operators that use assert(...)

Replace nv dependency with rocm software stack

• BLAS
• MIOpen
• CUB / Thrust
• Eigen -- need to find exactly what is needed.
• rocRAND
• rocFFT
• Sparse (?)
• Solver -- need to find exactly which APIs are required

Replace cub-hip with rocprim

We need to replace cub-hip with RocM official parallel primitive --> rocPRIM.

The pre-release can be found here:

Code: https://github.com/ROCmSoftwarePlatform/rocPRIM/tree/milestone3_prerelease
Release: https://github.com/ROCmSoftwarePlatform/rocPRIM/releases/tag/v0.2.1

Integrate rocblas and rocrand

Customer engagement (You Know Who)

specific workload tuning for the customers (Mask R-CNN etc)

Benchmark against NV (single node) / Operator Profiling

Running MIOpen BN throws warning message

Running SpatialBatchNorm op in MIOpen path generates a warning on the command line:

E0328 18:07:28.225716 127492 spatial_batch_norm_op_miopen.cc:38] Provided epsilon is smaller than MIOPEN_BN_MIN_EPSILON. Setting it to MIOPEN_BN_MIN_EPSILON instead.

This is happening because Caffe2 sets epsilon_ to 1e-5 which is same as MIOPEN_BN_MIN_EPSILON, which triggers the condition on ln 38 of spatial_batch_norm_op_miopen.cc
CUDNN uses condition if (epsilon_ <= CUDNN_BN_MIN_EPSILON - FLT_EPSILON)
instead of if (epsilon_ <= MIOPEN_BN_MIN_EPSILON)

Need to investigate what is proper value of MIOPEN_BN_MIN_EPSILON and the condition

Hipify operators with cuda implementation && no other dependencies and validate.

Total number of operators - 210

CUDA operators - 77 (some of underlying dependency on CUB/Thrust)
cuDNN operators - 11

Tasks includes but not limited to :

• hipify ops with cuda implementation (77 total)

• write python tests to validate the hip ops

fix the path for libcaffe2_hip.so for python based tests

when running python based tests:

WARNING:root:Debug message: ../lib/libcaffe2_hip.so: cannot open shared object file: No such file or directory

miopen ops looking for reference in libcaffe2.so

Not sure if this is env issue; need to investigate why this is happening (should go to libcaffe2_hip.so instead)

WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.
WARNING:root:Debug message: No module named caffe2_pybind11_state_gpu
CRITICAL:root:Cannot load caffe2.python. Error: /usr/local/lib/libcaffe2.so: undefined symbol: hipFree

rocm-caffe2 distributed mode

docker images: docker.io/rocm/caffe2:rocm1.7-miopen-dev-v1

For the single node, it is ok when used "python ../caffe2/python/examples/resnet50_trainer.py --train_data=/work/DL/dataset/cifar10/cifar10_train_lmdb/ --num_shards=1 --shard_id=0 --run_id=10 --file_store_path=/work/caffe2/". And the work directory is NFS shared .
But when I run two nodes I got some errors.
node1: python ../caffe2/python/examples/resnet50_trainer.py --train_data=/work/DL/dataset/cifar10/cifar10_train_lmdb/ --num_shards=2 --shard_id=0 --run_id=10 --file_store_path=/work/caffe2/
node2: python ../caffe2/python/examples/resnet50_trainer.py --train_data=/work/DL/dataset/cifar10/cifar10_train_lmdb/ --num_shards=2 --shard_id=1 --run_id=10 --file_store_path=/work/caffe2/

The outputs are as follows:
RuntimeError: [enforce fail at no_default_engine_op.h:45] . The operator CreateCommonWorld does not have a default engine implementation. Please specify an engine explicitly for this operator. Error from operator:
input: "store_handler" output: "allreduce_0_cw" name: "allreduce_0_cw_op" type: "CreateCommonWorld" arg { name: "timeout_ms" i: 30000 } arg { name: "rank" i: 0 } arg { name: "interface" s: "" } arg { name: "status_blob" s: "create_allreduce_cw_0_status" } arg { name: "transport" s: "tcp" } arg { name: "size" i: 2 } device_option { device_type: 4 hip_gpu_id: 0 } engine: "GLOO"

Thank you.

Long running time for python base op test

I am running relu_op test with MIOPEN as engine. Test is passing however running time varies each time.
range is roughly from 0.6 sec to 3.9 sec.

Need to investigate where this is issue in implementation or opportunity to improve the test infra.