rocm/rocm_bandwidth_test

Bandwidth test for ROCm

ROCm Bandwidth Test

Introduction

ROCmBandwidthTest is designed to capture the performance characteristics of buffer copying and kernel read/write operations. The help screen of the benchmark shows various options one can use in initiating cop/read/writer operations. In addition one can also query the topology of the system in terms of memory pools and their agents

Build Environment

To be able to build ROCmBandwidthTest, users must ensure that the build platform has following conditions satisfied:

Build Procedure

The following simply lists the steps to build ROCmBandwidthTest

--- Create a build directory in the project folder - roc_bandwidth_test.

mkdir ./build

--- Set working directory to be the new build directory

cd ./build

--- Invoke Cmake to interpret build rules and generate native build files The argument for cmake should be the root folder of ROCmBandwidthTest test suite

# Assume that ROCR Runtime has its libraries & headers are located in the path :
# libraries : _ABSOLUTE_PATH_TO_ROCR_LIBS_/lib
# headers   : _ABSOLUTE_PATH_TO_ROCR_LIBS_/include/hsa
# Note : Observe that both include & lib folder are under common path (_ABSOLUTE_PATH_TO_ROCR_LIBS_)

# Builds Debug version
# Assumes pwd is _ABSOLUTE_PATH_TO_RBT/roc_bandwidth_test/build

cmake -DCMAKE_BUILD_TYPE="Debug"   \
    -DCMAKE_MODULE_PATH="_ABSOLUTE_PATH_TO_RBT/rocm_bandwidth_test/cmake_modules"  \
    -DCMAKE_PREFIX_PATH="_ABSOLUTE_PATH_TO_ROCR_LIBS_"
    ..

# Builds Release version - default
# Assumes pwd is _ABSOLUTE_PATH_TO_RBT/roc_bandwidth_test/build
cmake -DCMAKE_MODULE_PATH="_ABSOLUTE_PATH_TO_RBT/rocm_bandwidth_test/cmake_modules"  \
    -DCMAKE_PREFIX_PATH="_ABSOLUTE_PATH_TO_ROCR_LIBS_"
    ..

--- Invoke the native build rules generated by cmake to build the various object, library and executable files

make

--- Invoke the install command to copy build artifacts to pre-defined folders of ROCmBandwidthTest suite. Upon completion artifacts will be copied to the bin and lib directories of build directory

make install

All executables will be found in <build_directory>/bin folder

rocm_bandwidth_test's Issues

Code Fails to Compile due to Warning about Mimatched Allocation Function

The code fails to compile due to the following warning that is being turned into an error via -Werror:

/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp: In member function ‘void RocmBandwidthTest::DisplayValidationMatrix() const’:
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp:352:7: error: ‘void free(void*)’ called on pointer returned from a mismatched allocation function [-Werror=mismatched-new-delete]
352 | free(perf_matrix);
| ~~~~^~~~~~~~~~~~~
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp:349:65: note: returned from ‘void* operator new ’
349 | double* perf_matrix = new doubleagent_index_ * agent_index_;
| ^
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp: In member function ‘void RocmBandwidthTest::DisplayCopyTimeMatrix(bool) const’:
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp:344:7: error: ‘void free(void*)’ called on pointer returned from a mismatched allocation function [-Werror=mismatched-new-delete]
344 | free(perf_matrix);
| ~~~~^~~~~~~~~~~~~
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp:341:65: note: returned from ‘void* operator new ’
341 | double* perf_matrix = new doubleagent_index_ * agent_index_;
| ^
In member function ‘void RocmBandwidthTest::DisplayValidationMatrix() const’,
inlined from ‘virtual void RocmBandwidthTest::Display() const’ at /home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp:165:28:
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp:352:7: error: ‘void free(void*)’ called on pointer returned from a mismatched allocation function [-Werror=mismatched-new-delete]
352 | free(perf_matrix);
| ~~~~^~~~~~~~~~~~~
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp: In member function ‘virtual void RocmBandwidthTest::Display() const’:
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp:349:65: note: returned from ‘void* operator new ’
349 | double* perf_matrix = new doubleagent_index_ * agent_index_;
| ^
In member function ‘void RocmBandwidthTest::DisplayCopyTimeMatrix(bool) const’,
inlined from ‘virtual void RocmBandwidthTest::Display() const’ at /home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp:174:26:
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp:344:7: error: ‘void free(void*)’ called on pointer returned from a mismatched allocation function [-Werror=mismatched-new-delete]
344 | free(perf_matrix);
| ~~~~^~~~~~~~~~~~~
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp: In member function ‘virtual void RocmBandwidthTest::Display() const’:
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp:341:65: note: returned from ‘void* operator new ’
341 | double* perf_matrix = new doubleagent_index_ * agent_index_;
| ^
In member function ‘void RocmBandwidthTest::DisplayCopyTimeMatrix(bool) const’,
inlined from ‘virtual void RocmBandwidthTest::Display() const’ at /home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp:185:26:
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp:344:7: error: ‘void free(void*)’ called on pointer returned from a mismatched allocation function [-Werror=mismatched-new-delete]
344 | free(perf_matrix);
| ~~~~^~~~~~~~~~~~~
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp: In member function ‘virtual void RocmBandwidthTest::Display() const’:
/home/micklemm/projects/rocm/rocm_bandwidth_test/rocm_bandwidth_test_report.cpp:341:65: note: returned from ‘void* operator new ’
341 | double* perf_matrix = new doubleagent_index_ * agent_index_;
| ^
cc1plus: all warnings being treated as errors
make[2]: *** [CMakeFiles/rocm-bandwidth-test.dir/build.make:189: CMakeFiles/rocm-bandwidth-test.dir/rocm_bandwidth_test_report.cpp.o] Error 1
make[2]: Leaving directory '/home/micklemm/projects/rocm/rocm_bandwidth_test'
make[1]: *** [CMakeFiles/Makefile2:98: CMakeFiles/rocm-bandwidth-test.dir/all] Error 2
make[1]: Leaving directory '/home/micklemm/projects/rocm/rocm_bandwidth_test'
make: *** [Makefile:174: all] Error 2
nice make 3,43s user 0,29s system 99% cpu 3,728 total
mun-gpu01 ~/p*/r*/rocm_bandwidth_test [0:0] ((rocm-4.2.0))> git checkout master
Switched to branch 'master'
Your branch is up to date with 'origin/master'.

$ /usr/bin/c++ --version
c++ (Ubuntu 11-20210417-1ubuntu1) 11.0.1 20210417 (experimental) [master revision c1c86ab96c2:b6fb0ccbb48:8ae884c09fbba91e9cec391290ee4a2859e7ff41]
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Incorrect (missing) dependencies in deb package

# apt-cache show rocm_bandwidth_test | egrep 'Package|Depend'
Package: rocm_bandwidth_test
#

# /opt/rocm/bin/rocm_bandwidth_test 
/opt/rocm/bin/rocm_bandwidth_test: error while loading shared libraries: libhsa-runtime64.so.1: cannot open shared object file: No such file or directory
#

# ldd /opt/rocm/bin/rocm_bandwidth_test 
	linux-vdso.so.1 (0x00007ffc26c1d000)
	libhsa-runtime64.so.1 => not found
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f89df647000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f89df63d000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f89df4a9000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f89df2ec000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f89df2d2000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f89df7f3000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f89df2af000)

Indicates it should depend on libc6, libstdc++6, libgcc1 and hsakmt-roct-dev.

Fix freeing of local array

build error

rocm_bandwidth_test_topology.cpp:296:63: error: ‘hsa_amd_memory_pool_link_info_t {aka struct hsa_amd_memory_pool_link_info_s}’ has no member named ‘numa_distance’
link_matrix_[(idx1 *agent_index_) + idx2] = link_info.numa_distance;

Compile failing

With Latest commit be6a8ba Aug 1, 2019

Scanning dependencies of target rocm_bandwidth_test
[  7%] Building CXX object CMakeFiles/rocm_bandwidth_test.dir/base_test.cpp.o
[ 14%] Building CXX object CMakeFiles/rocm_bandwidth_test.dir/common.cpp.o
[ 21%] Building CXX object CMakeFiles/rocm_bandwidth_test.dir/hsatimer.cpp.o
[ 28%] Building CXX object CMakeFiles/rocm_bandwidth_test.dir/main.cpp.o
[ 35%] Building CXX object CMakeFiles/rocm_bandwidth_test.dir/os.cpp.o
[ 42%] Building CXX object CMakeFiles/rocm_bandwidth_test.dir/rocm_bandwidth_test.cpp.o
[ 50%] Building CXX object CMakeFiles/rocm_bandwidth_test.dir/rocm_bandwidth_test_io.cpp.o
[ 57%] Building CXX object CMakeFiles/rocm_bandwidth_test.dir/rocm_bandwidth_test_parse.cpp.o
[ 64%] Building CXX object CMakeFiles/rocm_bandwidth_test.dir/rocm_bandwidth_test_print.cpp.o
/home/amdfanboy/github/rocm_bandwidth_test/rocm_bandwidth_test_print.cpp: In function ‘std::__cxx11::string GetValueAsString(uint32_t, uint32_t)’:
/home/amdfanboy/github/rocm_bandwidth_test/rocm_bandwidth_test_print.cpp:181:21: error: control reaches end of non-void function [-Werror=return-type]
   std::stringstream ss;
                     ^~
At global scope:
cc1plus: error: unrecognized command line option ‘-Wno-dev’ [-Werror]
cc1plus: all warnings being treated as errors
make[2]: *** [CMakeFiles/rocm_bandwidth_test.dir/build.make:167: CMakeFiles/rocm_bandwidth_test.dir/rocm_bandwidth_test_print.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:73: CMakeFiles/rocm_bandwidth_test.dir/all] Error 2
make: *** [Makefile:152: all] Error 2

rocm-bandwidth-test -v command doesn't work

# ./rocm-bandwidth-test  -v 

Supported arguments:

         -h    Prints the help screen
         -q    Query version of the test
         -v    Run the test in validation mode
         -l    Run test to collect Latency data
         -c    Time the operation using CPU Timers
         -i    Initialize copy buffer with specified 'long double' pattern
         -t    Prints system topology and allocatable memory info
         -m    List of buffer sizes to use, specified in Megabytes
         -b    List devices to use in bidirectional copy operations
         -s    List of source devices to use in copy unidirectional operations
         -d    List of destination devices to use in unidirectional copy operations
         -a    Perform Unidirectional Copy involving all device combinations
         -A    Perform Bidirectional Copy involving all device combinations

         NOTE: Mixing following options is illegal/unsupported
                 Case 1: rocm_bandwidth_test -a with {lm}{1,}
                 Case 2: rocm_bandwidth_test -b with {clv}{1,}
                 Case 3: rocm_bandwidth_test -A with {clmv}{1,}
                 Case 4: rocm_bandwidth_test -s x -d y with {lmv}{2,}

rocm_bandwdith_test can't communicate gpu with gpu

I have a setting where there are 8 CPUs (Intel) and two GPUs, ran the rocm_bandwidth_test, and couldn't measure cross GPU communication

I used rocm 1.9 rpms from http://repo.radeon.com/rocm/misc/facebook/yum/.yum_1.9.white_rabbit/
this is on centos 7.5.1804 with kernel 3.10.0-862.el7.x86_64

Typo: "sprcified" should be "specified"

https://github.com/RadeonOpenCompute/rocm_bandwidth_test/blob/c765fd3e42ec7b21f5a928fc993b5b3ab39fce30/CMakeLists.txt#L51

Infinity Fabric for GPU-GPU data transfers?

There is no call of hipDeviceEnablePeerAccess in this benchmark. Does it mean that with the current version there is no possibility to test Infinity Fabric for GPU-GPU data transfers?

rocm_bandwidth_test deb package should not use underscores.

It is somehow a bad practice in Debian packaging world to use underscores in package names and package file names.

Consider rocm-bandwidthtest

Incorrect path in CMake instructions in Readme.md

For the two cmake examples, there is an error in the path.

This should be -DCMAKE_MODULE_PATH="_ABSOLUTE_PATH_TO_RBT/rocm_bandwidth_test/cmake_modules" rather than
-DCMAKE_MODULE_PATH="_ABSOLUTE_PATH_TO_RBT/roc_bandwidth_test/cmake_modules"