microsoft/diskspd-for-linux

A disk io load-generator and benchmarking tool for Linux, based on the Windows tool diskspd.

diskspd-for-linux

Summary

A disk io load-generator and benchmarking tool for Linux, based on the Windows tool diskspd.

Features

• Easy to read output showing IOPS, throughput (MB/s), average latency (-L) and standard deviations (-L -D)
• Multiple threads doing work on the same file or different files, various access patterns available (-t -F -s -T)
• CPU affinity - specify a set of CPU's to bind threads to (-a -n)
• Overlapped (async) io with choice of libaio (linux kernel aio) or posix aio (userspace threads) (-x)

Getting Started

Dependencies and System Requirements

• make
• gcc 5.4 or newer
• libaio-dev

Building and installing

Build

    make

Install (to /usr/local/bin)

    make install

Debug build (enables debugging output with d_printf() function):

    make DEBUG=1

You may need to make clean first if you already built diskspd without debugging turned on

Static linking (enables static libraries for binary portability):

    make STATIC=1

Usage

    diskspd [OPTIONS...] FILE [FILE...]

At least one file (or disk) must be specified, along with a number of optional arguments. Don't leave a space between the option and its argument; some arguments are optional and argp doesn't support spaces in this case. To have diskspd create files of a specified size, use -c followed by the file size.

To see the full list of options, use:

    diskspd --help

Examples and tips

By default, diskspd creates a single thread for each file or disk specified and does sequential IO operations starting at offset 0, keeping at most 2 operations in flight at a time per-thread. These operations are offset from each other by the stride (which defaults to the specified block or IO size, which itself defaults to 64K).

The most basic usage of the tool is to simply use -c to create a file to do IO on, and run with the defaults options enabled. This example creates a 1MiB file filled with a repeating pattern and reads from it sequentially until the test is done. The Linux kernel's asynchronous io interface is used to keep at most two operations in flight. A 5 second warmup is followed by 10 seconds of recording results.

    diskspd -c1M testfile

The following example writes directly (without buffering) to the partition /dev/sdc1 without touching the first 1GiB of the disk, using 1MiB blocks. Warm up is only 2 seconds, and results are recorded for 60 seconds. Output latency information and standard deviation for IOPS are displayed. 8 threads are used in total each keeping at most 4 operations in flight at a time, for a total of up to 32 in flight operations at a time.

    diskspd -b1M -B1G -w100 -Sh -W2 -d60 -L -D -t8 -o4 /dev/sdc1

Useful options:

• -F and -t set the total number of threads to use. Mutually exclusive with -t

• -d modifies the duration of the test (default is 10 seconds)

• -Sh opens files with O_DIRECT and O_SYNC flags, ensuring no caching is done in the OS. Note that all I/O must be aligned to the underlying device's physical sector size for this option to work.

• -t modifies how many threads do work on each file (default is 1)

• -W modifies the warmup time (default is 5 seconds)

• -z uses C++'s random_device to seed the offsets generated by -r (default is a seed of 0)

• -Zr uses separate read and write I/O buffers, and fill them with random data (default is ascending bytes)

More detailed information about diskspd's usage can be found in the wiki on GitHub.

Planned features

• Cooldown time (-C), same as warmup time but at the end of the duration
• fadvise flags to hint at file access patterns (analogous to Windows diskspd's -f<rst>)
• -h alternative for printing help
• I/O burst size (-i) and pause (-j). Issue bursts of IO with some milliseconds of pause in between them
• Use large pages for buffers (-l)
• JSON/XML input (-[XJ] FILE) and output (-R[jx]) (to file or just stdout)
  • specify most options at a per-target level using a JSON or XML file
  • output results to JSON or XML, possibly including more data (e.g. latency histogram data)
• Unit tests
• Validation/verification mode. Currently diskspd for linux errors out if bytes read/written aren't equal to block size (this only happens when there are alignment issues and such).

Known issues

• File setup quite slow for the first run
  • currently mitigated by simply using existing files if they're of the right size or larger
  • parallelizing file setup would improve this
• Some combinations of inputs may cause undefined behaviour
  • for the most part everything is checked for validity and conflicting options error out or are ignored, but it's possible not every combination is covered
• Possible bugs with Iops std-dev calculation - it seems to match diskspd for Windows behaviour however. This could be an issue with how the IoBucketizer is used, or something else.

Terms of Use

By downloading and running this project, you agree to the license terms of the third party application software, Microsoft products, and components to be installed.

The third party software and products are provided to you by third parties. You are responsible for reading and accepting the relevant license terms for all software that will be installed. Microsoft grants you no rights to third party software.

License

Copyright (c) 2018 Microsoft Corporation. All rights reserved.

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

diskspd-for-linux's Issues

not work on Ubuntu 20.04.3

I compled latest source on Ubuntu 20.04.3.
execute diskspd error with non specific device.

$ sudo diskspd -v -c100M testfile
Unexpected error '2: No such file or directory' when statting /dev/nvme0c0n1!
stat: No such file or directory
$ sudo diskspd -v -d60 -b4K -r -w0 -L /dev/nvme1n1
Unexpected error '2: No such file or directory' when statting /dev/nvme0c0n1!
stat: No such file or directory
$ sudo diskspd -v -d60 -b4K -r -w0 -L /dev/nvme1
Unexpected error '2: No such file or directory' when statting /dev/nvme0c0n1!
stat: No such file or directory
$

/dev/nvme0c0n1 not foud.
why /dev/nvme0c0n1 ?

$ ls -l /dev/nvme*
crw------- 1 root root 239, 0 Dec 21 17:43 /dev/nvme0
brw-rw---- 1 root disk 259, 1 Dec 21 17:43 /dev/nvme0n1
crw------- 1 root root 239, 1 Dec 21 17:43 /dev/nvme1
brw-rw---- 1 root disk 259, 3 Dec 21 17:43 /dev/nvme1n1
$ ls -ld /sys/block/nvme*
lrwxrwxrwx 1 root root 0 Dec 21 17:47 /sys/block/nvme0c0n1 -> ../devices/pci0000:64/0000:64:02.0/0000:65:00.0/nvme/nvme0/nvme0c0n1
lrwxrwxrwx 1 root root 0 Dec 21 17:47 /sys/block/nvme0n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1
lrwxrwxrwx 1 root root 0 Dec 21 17:47 /sys/block/nvme1c1n1 -> ../devices/pci0000:64/0000:64:03.0/0000:66:00.0/nvme/nvme1/nvme1c1n1
lrwxrwxrwx 1 root root 0 Dec 21 17:47 /sys/block/nvme1n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1
$

Ubuntu 20.04.3 LTS
kernel 5.4.0-91-generic #102-Ubuntu
gcc 9.3.0-17
g++ 9.3.0-17

Compile error: undefined reference to `__gthrw___pthread_key_create

This is an error I got trying to compile on Centos 7.5. After looking it up, this link helped me out:

https://stackoverflow.com/questions/27257655/undefined-reference-to-gthrw-pthread-key-createunsigned-int-void-voi

As it recommends, putting "include " in options.cc resolved the issue.
Figured I would pass this along to you.

There were other errors, too:

src/posix_aio.cc:251:19: error: no matching function for call to ‘std::vector<aiocb*>::erase(__gnu_cxx::__normal_iterator<aiocb* const*, std::vector<aiocb*> >&)’
vec->erase(it);

and:

src/result_formatter.cc: In member function ‘virtual void diskspd::ResultFormatterText::output_results(const diskspd::Profile&)’:
src/result_formatter.cc:107:48: error: variable-sized object ‘cpu_usage_totals’ may not be initialized
double cpu_usage_totals[num_cpu_fields] = {0};

But I was able to work around them with hacks...
Thanks,
-mike

Some Issues

• Testfile is written with O_SYNC, without O_SYNC but with fsync at the end it could be faster.
• If Testfile was not written in Time, diskspd will exit, but running along in the Background.
  This could happen on machines with slow IO.
• g++ complains about "reaches end of control without return in non-void function".

Observing lower throughput compared to FIO

4K Performance drop -
20% - Random Read
50% - Random Write
40% - Seq Read
60% - Seq Write
Diskspd logs:
diskspdLog.zip

Tried to lookup nonexistent device X,Y in sys_info!

If I run diskspd (latest source) on a ZFS partition it does not work, I get this message 'Tried to lookup nonexistent device 0,37 in sys_info!'

On the same system but another partition (EXT) it works.

doesn't work with bcache

Ubuntu 20.04.4 LTS
Linux vcc-ldc-vs-10 5.4.0-113-generic #127-Ubuntu SMP Wed May 18 14:30:56 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Trying to run diskspd on a bcache device or mounted filesystem always results in the same error:

root@vcc-ldc-vs-10:~# df /mnt/tank/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/bcache0 125020192760 871697380 124148495380 1% /mnt/tank

root@vcc-ldc-vs-10:~# diskspd -c8G -b512K -w50 -Sh -d60 -r4K /mnt/tank/testfile-delete.dat
ret from aio not equal to block size, it's -22
Error during warmup phase!
Job 0 failed, exiting

root@vcc-ldc-vs-10:~# umount /mnt/tank

root@vcc-ldc-vs-10:~# diskspd -b1M -w50 -Sh -d60 -r4K /dev/bcache0
ret from aio not equal to block size, it's -22
Error during thread initialization!
Job 0 failed, exiting

When writing to a filesystem, as in the former example, the test file is written before the error is displayed. When writing to the raw device the error is displayed after less than a second. The same commands work when run against the backing device only, or to the filesystem when only the backing device is mounted, but not to the bcache device or filesystem mounted thereon.

edited for better formatting and clarity.

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Error on 'make'

src/kernel_aio.cc:14:10: fatal error: libaio.h: No such file or directory
#include <libaio.h>
^~~~~~~~~~
compilation terminated.
Makefile:59: recipe for target 'src/kernel_aio.o' failed
make: *** [src/kernel_aio.o] Error 1

diskspd on BTRFS not working

Hello,

if I run diskspd on a BTRFS filesystem i get the following error:

root@server02:/btrfstest/source# /usr/src/diskspd-for-linux/bin/diskspd -b8K -d30 -o4 -t8 -r -w25 -c1G -L -v test.perf
Setting up target files
Tried to lookup nonexistent device 0,96 in sys_info!
root@server02:/btrfstest/source#

root@server02:/btrfstest/source# mount | grep -v docker| grep -v cgroup
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1989148k,nr_inodes=497287,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=403912k,mode=755)
/dev/mapper/yii2--01--vg-root on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=352)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
/dev/vda1 on /boot type ext2 (rw,relatime,block_validity,barrier,user_xattr,acl)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=403908k,mode=700,uid=1000,gid=1000)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=403908k,mode=700)
/dev/vdb on /btrfstest/source type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
/dev/vdc on /btrfstest/dest type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)

Version #?

Thanks for porting this valuable utility!

I couldn't find any versioning-- do we know approximately what Windows version this Linux port corresponds to?

Even if it doesn't directly correspond to a Windows version, it probably should have some versioning in it...

.deb package

It'd be great if this could be deployed via the other Debian package repositories Microsoft has, such as for VS Code and .NET Core.

For example:
https://packages.microsoft.com/repos/microsoft-debian-buster-prod

nvme disk device benchmark does not work

Trying to execute a benchmark directly on a nvme device does not work.

$ sudo ./bin/diskspd -d60 -b4K -r -w0 -L /dev/nvme0n1
Couldn't open scheduler file for device nvme0

$ sudo ./bin/diskspd -d60 -b4K -r -w0 -L /dev/nvme0  
Tried to lookup nonexistent device 239,0 in sys_info!

The first error comes from src/sys_info.cc where /sys/block/nvme0/queue/scheduler should be opened but does not exist:

        std::string SysInfo::scheduler_from_device(std::string device) {
 
                std::string line;
                std::ifstream schedfile(std::string("/sys/block/"+device+"/queue/scheduler"));

Manually changing the code to use nvme0n1 instead of nvme0 leads to a successful benchmark run. I guess the logic for stripping the partition of a block device does not work for nvme and must be adapted.

The structure under /sys/block looks like the following:

$ ls -hl /sys/block
total 0
lrwxrwxrwx. 1 root root 0 Nov 22 11:22 dm-0 -> ../devices/virtual/block/dm-0
lrwxrwxrwx. 1 root root 0 Nov 24 12:03 loop0 -> ../devices/virtual/block/loop0
...
lrwxrwxrwx. 1 root root 0 Nov 24 09:02 nvme0n1 -> ../devices/pci0000:00/0000:00:1d.4/0000:71:00.0/nvme/nvme0/nvme0n1
lrwxrwxrwx. 1 root root 0 Nov 22 08:54 zram0 -> ../devices/virtual/block/zram0