ublk-org/ublksrv

ublk: userspace block device driver

ublksrv's Issues

README is missing references.

The README currently references a number of external links but only one reference is actually given?

e.g. "Also looks linux-mm can't support zero copy for this case yet[3]."

failed to create ublk device with qcow2

I can not create ublk device with qcow2,

# qemu-img create -f qcow2 ublk_qcow2.qcow2 10G
Formatting 'ublk_qcow2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=10737418240 lazy_refcounts=off refcount_bits=16
# ublk add -t qcow2 -f ublk_qcow2.qcow2
start dev 0 daemon failed, ret -1
#
# ublk -v
ublksrv 1.1-10-g6550517

and the testing generic/003 ,generic/005 all hit this issue:

running generic/003
	mount/umount /dev/ublkb0(ublk -t loop -q 2 -g 0 -f /mnt/tests/gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/archive/main/kernel-tests-main.zip/storage/block/RHEL_2107_ublk_testing/ubdsrv/tests/tmp/ublk_loop_null_1G_P1zCV) with fstype(xfs)...ok
	mount/umount /dev/ublkb0(ublk -t loop -q 2 -g 1 -f /mnt/tests/gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/archive/main/kernel-tests-main.zip/storage/block/RHEL_2107_ublk_testing/ubdsrv/tests/tmp/ublk_loop_null_1G_P1zCV) with fstype(xfs)...ok
	mount/umount /dev/ublkb0(ublk -t qcow2 -q 1 -g 0 -f /mnt/tests/gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/archive/main/kernel-tests-main.zip/storage/block/RHEL_2107_ublk_testing/ubdsrv/tests/tmp/ublk_null_1G_HA4aN.qcow2) with fstype(xfs)...failed
	mount/umount /dev/ublkb0(ublk -t qcow2 -q 1 -g 1 -f /mnt/tests/gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/archive/main/kernel-tests-main.zip/storage/block/RHEL_2107_ublk_testing/ubdsrv/tests/tmp/ublk_null_1G_HA4aN.qcow2) with fstype(xfs)...failed
	mount/umount /dev/ublkb0(ublk -t nbd -q 1 -g 0 -d 127 --host 127.0.0.1) with fstype(xfs)...failed
	mount/umount /dev/ublkb0(ublk -t nbd -q 1 -g 1 -d 127 --host 127.0.0.1) with fstype(xfs)...failed

running generic/005
	run fio with dev recovery, type 1:
	(1)kill all ubq_deamon, (2)recover with new ubq_daemon, (3)delete dev
	run fio with killing /dev/ublkb0(ublk add -t null -q 2 -u 1 -g 1 -r 1 -i 0 ) queue daemon 1
	run fio with killing /dev/ublkb0(ublk add -t null -q 2 -u 1 -g 1 -r 1 -i 1 ) queue daemon 1
	run fio with killing /dev/ublkb0(ublk add -t loop -q 2 -u 1 -g 1 -r 1 -i 0 -f /mnt/tests/gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/archive/main/kernel-tests-main.zip/storage/block/RHEL_2107_ublk_testing/ubdsrv/tests/tmp/ublk_loop_null_256M_7rATm) queue daemon 1
	run fio with killing /dev/ublkb0(ublk add -t loop -q 2 -u 1 -g 1 -r 1 -i 1 -f /mnt/tests/gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/archive/main/kernel-tests-main.zip/storage/block/RHEL_2107_ublk_testing/ubdsrv/tests/tmp/ublk_loop_null_256M_RQNoP) queue daemon 1
	run fio with killing /dev/ublkb0(ublk add -t qcow2 -q 1 -u 1 -g 1 -r 1 -i 0 -f /mnt/tests/gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/archive/main/kernel-tests-main.zip/storage/block/RHEL_2107_ublk_testing/ubdsrv/tests/tmp/ublk_null_256M_qWWtG.qcow2) queue daemon 1
can't get dev info from 0: -19
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
can't get dev info from 0: -19
can't get dev info from 0: -19
can't get dev info from 0: -19
can't get dev info from 0: -19
can't get dev info from 0: -19
can't get dev info from 0: -19
can't get dev info from 0: -19
can't get dev info from 0: -19
can't get dev info from 0: -19
can't get dev info from 0: -19
can't get dev info from 0
can't get dev info from 0
can't get dev info from 0

ubdsrv does 'make' failed with no IOURING_OP_URING_CMD undeclared

Error msg as below:

make -C /nvmeStor/ubdsrv/lib
make[1]: Entering directory '/nvmeStor/ubdsrv/lib'
gcc -c -g -O2 -D_GNU_SOURCE -MMD -fPIC -I ../include ublksrv_cmd.c -o ublksrv_cmd.o
In file included from ublksrv_cmd.c:3:
../include/ublksrv_priv.h: In function ‘ublksrv_get_sqe_cmd’:
../include/ublksrv_priv.h:79:30: error: ‘struct io_uring_sqe’ has no member named ‘addr3’; did you mean ‘addr2’?
   79 |         return (void *)&sqe->addr3;
      |                              ^~~~~
      |                              addr2
ublksrv_cmd.c: In function ‘ublksrv_ctrl_init_cmd’:
ublksrv_cmd.c:28:23: error: ‘IORING_OP_URING_CMD’ undeclared (first use in this function)
   28 |         sqe->opcode = IORING_OP_URING_CMD;
      |                       ^~~~~~~~~~~~~~~~~~~
ublksrv_cmd.c:28:23: note: each undeclared identifier is reported only once for each function it appears in
ublksrv_cmd.c: In function ‘ublksrv_ctrl_init’:
ublksrv_cmd.c:121:50: error: ‘IORING_SETUP_SQE128’ undeclared (first use in this function); did you mean ‘IORING_SETUP_SQ_AFF’?
  121 |         ret = ublksrv_setup_ring(32, &dev->ring, IORING_SETUP_SQE128);
      |                                                  ^~~~~~~~~~~~~~~~~~~
      |                                                  IORING_SETUP_SQ_AFF
make[1]: *** [Makefile:16: ublksrv_cmd.o] Error 1
make[1]: Leaving directory '/nvmeStor/ubdsrv/lib'
make: *** [Makefile:23: lib/libublksrv.a] Error 2

ENV:

[root@fedora ubdsrv]# rpm -qa |grep uring
liburing-2.0-3.fc36.x86_64
liburing-devel-2.0-3.fc36.x86_64
[root@fedora ubdsrv]# uname -a
Linux fedora 5.19.0-rc4+ #2 SMP PREEMPT_DYNAMIC Tue Aug 23 04:45:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Will be v1.1 released?

v1.1-rc1 is released at 2023-02-04. Will v1.1 be released anytime soon?

ps. I am considering if this worth to build ubdsrv package for ALT Linux distributions.
See who currently builds it https://repology.org/project/ubdsrv/versions
Distributions rarely build form master or random commit and usually build from a release tag.

[build error] undefined reference to __find_slice()

Environment:
I' m running a Debian 11 Virtual Machine, and I have installed the latest 6.0 Linux kernel.

Problem:
I' m following the official instructions from README, (btw I think we need to add explicit instructions prior to autoreconf -i and ./configure, because they need some packages to be installed. I can do it in a subsequent PR if you agree) and I'm bumping into the following problem:

$ make
make  all-recursive
make[1]: Entering directory '/home/dim/ubdsrv'
Making all in include
make[2]: Entering directory '/home/dim/ubdsrv/include'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/dim/ubdsrv/include'
Making all in lib
make[2]: Entering directory '/home/dim/ubdsrv/lib'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/dim/ubdsrv/lib'
Making all in tests
make[2]: Entering directory '/home/dim/ubdsrv/tests'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/dim/ubdsrv/tests'
make[2]: Entering directory '/home/dim/ubdsrv'
/bin/bash ./libtool  --tag=CXX   --mode=link g++ -fcoroutines -std=c++20 -g -O2 -pthread   -o ublk ublk-ublksrv_tgt.o ublk-tgt_null.o ublk-tgt_loop.o ublk-tgt_qcow2.o ublk-qcow2.o ublk-qcow2_meta.o ublk-utils.o lib/libublksrv.la -luring  
libtool: link: g++ -fcoroutines -std=c++20 -g -O2 -pthread -o .libs/ublk ublk-ublksrv_tgt.o ublk-tgt_null.o ublk-tgt_loop.o ublk-tgt_qcow2.o ublk-qcow2.o ublk-qcow2_meta.o ublk-utils.o  lib/.libs/libublksrv.so -luring -pthread
/usr/bin/ld: ublk-qcow2_meta.o: in function `Qcow2ClusterMapping::__find_slice(unsigned long, bool)':
/home/dim/ubdsrv/qcow2/qcow2.h:1181: undefined reference to `slice_cache<Qcow2L2Table>::__find_slice(unsigned long, bool)'
/usr/bin/ld: ublk-qcow2_meta.o: in function `Qcow2ClusterAllocator::__find_slice(unsigned long)':
/home/dim/ubdsrv/qcow2/qcow2.h:1292: undefined reference to `slice_cache<Qcow2RefcountBlock>::__find_slice(unsigned long, bool)'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:600: ublk] Error 1
make[2]: Leaving directory '/home/dim/ubdsrv'
make[1]: *** [Makefile:829: all-recursive] Error 1
make[1]: Leaving directory '/home/dim/ubdsrv'
make: *** [Makefile:477: all] Error 2

Note: If I copy the body of __find_slice() (as defined in qcow2/qcow2.cpp), under the declaration of __find_slice() in qcow2/qcow2.h, the error is gone.

Having said that, I just want to inform you that prior to this, I faced during make the 2 following errors:

1. Same as issue #20.
   I solved it, by including the <sys/time.h> in qcow2/qcow2.h.
2. Undefined reference to various pthread specific functions (ptread_create(), pthread_join() etc). I solved it, by appending the -pthread flag to CXXFLAGS in Makefile.

ublksrv_ctrl_del_dev block

Call Trace:
[ 2780.096864]
[ 2780.096866] __schedule+0x2aa/0x610
[ 2780.096868] schedule+0x63/0x110
[ 2780.096873] ublk_ctrl_uring_cmd+0x12b4/0x146f [ublk_drv]
[ 2780.096877] ? __pfx_autoremove_wake_function+0x10/0x10
[ 2780.096880] io_uring_cmd+0x9d/0x140
[ 2780.096883] io_issue_sqe+0x1b2/0x410
[ 2780.096884] io_wq_submit_work+0x8d/0x2c0
[ 2780.096886] io_worker_handle_work+0x114/0x320
[ 2780.096888] io_wqe_worker+0x132/0x380
[ 2780.096891] ? raw_spin_rq_unlock+0x10/0x40
[ 2780.096893] ? finish_task_switch.isra.0+0x85/0x2a0
[ 2780.096895] ? __pfx_io_wqe_worker+0x10/0x10
[ 2780.096897] ret_from_fork+0x2c/0x50

io_uring=2.3 liburing=2.4 call ublksrv_ctrl_del_dev block,

cannot link to pthread

[root@iZuf609uyxtf9dw6jr2adgZ ubdsrv]# autoreconf -i
[root@iZuf609uyxtf9dw6jr2adgZ ubdsrv]# export LIBURING_DIR=/work/liburing-liburing-2.2; PKG_CONFIG_PATH=${LIBURING_DIR} ./configure CFLAGS="-I${LIBURING_DIR}/src/include" CXXFLAGS="-I${LIBURING_DIR}/src/include" LDFLAGS="-L${LIBURING_DIR}/src"
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define EXTENSIONS... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports the include directive... yes (GNU style)
checking whether make supports nested variables... yes
checking dependency style of gcc... gcc3
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking how to print strings... printf
checking for a sed that does not truncate output... /usr/bin/sed
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @file support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /usr/bin/dd
checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
checking for mt... no
checking if : is a manifest tool... no
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for a sed that does not truncate output... (cached) /usr/bin/sed
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking how to run the C preprocessor... gcc -E
checking for function prototypes... yes
checking whether gcc is Clang... no
checking whether pthreads work with -pthread... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking whether more special flags are required for pthreads... no
checking for PTHREAD_PRIO_INHERIT... yes
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... no
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for LIBURING... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating include/Makefile
config.status: creating lib/Makefile
config.status: creating tests/Makefile
config.status: creating ublksrv.pc
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
config.status: executing libtool commands
[root@iZuf609uyxtf9dw6jr2adgZ ubdsrv]# make clean && make
Making clean in include
make[1]: Entering directory '/root/work/ubdsrv/include'
rm -rf .libs _libs
rm -f .lo
make[1]: Leaving directory '/root/work/ubdsrv/include'
Making clean in lib
make[1]: Entering directory '/root/work/ubdsrv/lib'
test -z " .d" || rm -f *~ .d
test -z "libublksrv.la" || rm -f libublksrv.la
rm -f ./so_locations
rm -rf .libs _libs
rm -f .o
rm -f .lo
make[1]: Leaving directory '/root/work/ubdsrv/lib'
Making clean in tests
make[1]: Entering directory '/root/work/ubdsrv/tests'
test -z "~ /" || rm -f * /~
rm -rf .libs _libs
rm -f .lo
make[1]: Leaving directory '/root/work/ubdsrv/tests'
make[1]: Entering directory '/root/work/ubdsrv'
rm -f ublk demo_null demo_event
test -z "~ test cscope. include/~ .d" || rm -f *~ test cscope. include/~ *.d
rm -rf .libs _libs
rm -f *.o
rm -f *.lo
make[1]: Leaving directory '/root/work/ubdsrv'
make all-recursive
make[1]: Entering directory '/root/work/ubdsrv'
Making all in include
make[2]: Entering directory '/root/work/ubdsrv/include'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/root/work/ubdsrv/include'
Making all in lib
make[2]: Entering directory '/root/work/ubdsrv/lib'
/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -I../include -pthread -I/root/work/liburing-liburing-2.2/src/include -MT libublksrv_la-ublksrv_cmd.lo -MD -MP -MF .deps/libublksrv_la-ublksrv_cmd.Tpo -c -o libublksrv_la-ublksrv_cmd.lo test -f 'ublksrv_cmd.c' || echo './'ublksrv_cmd.c
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -pthread -I/root/work/liburing-liburing-2.2/src/include -MT libublksrv_la-ublksrv_cmd.lo -MD -MP -MF .deps/libublksrv_la-ublksrv_cmd.Tpo -c ublksrv_cmd.c -fPIC -DPIC -o .libs/libublksrv_la-ublksrv_cmd.o
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -pthread -I/root/work/liburing-liburing-2.2/src/include -MT libublksrv_la-ublksrv_cmd.lo -MD -MP -MF .deps/libublksrv_la-ublksrv_cmd.Tpo -c ublksrv_cmd.c -o libublksrv_la-ublksrv_cmd.o >/dev/null 2>&1
mv -f .deps/libublksrv_la-ublksrv_cmd.Tpo .deps/libublksrv_la-ublksrv_cmd.Plo
/bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -I../include -I/root/work/liburing-liburing-2.2/src/include -MT libublksrv_la-ublksrv_json.lo -MD -MP -MF .deps/libublksrv_la-ublksrv_json.Tpo -c -o libublksrv_la-ublksrv_json.lo test -f 'ublksrv_json.cpp' || echo './'ublksrv_json.cpp
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -I../include -I/root/work/liburing-liburing-2.2/src/include -MT libublksrv_la-ublksrv_json.lo -MD -MP -MF .deps/libublksrv_la-ublksrv_json.Tpo -c ublksrv_json.cpp -fPIC -DPIC -o .libs/libublksrv_la-ublksrv_json.o
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -I../include -I/root/work/liburing-liburing-2.2/src/include -MT libublksrv_la-ublksrv_json.lo -MD -MP -MF .deps/libublksrv_la-ublksrv_json.Tpo -c ublksrv_json.cpp -o libublksrv_la-ublksrv_json.o >/dev/null 2>&1
mv -f .deps/libublksrv_la-ublksrv_json.Tpo .deps/libublksrv_la-ublksrv_json.Plo
/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -I../include -pthread -I/root/work/liburing-liburing-2.2/src/include -MT libublksrv_la-ublksrv.lo -MD -MP -MF .deps/libublksrv_la-ublksrv.Tpo -c -o libublksrv_la-ublksrv.lo test -f 'ublksrv.c' || echo './'ublksrv.c
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -pthread -I/root/work/liburing-liburing-2.2/src/include -MT libublksrv_la-ublksrv.lo -MD -MP -MF .deps/libublksrv_la-ublksrv.Tpo -c ublksrv.c -fPIC -DPIC -o .libs/libublksrv_la-ublksrv.o
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -pthread -I/root/work/liburing-liburing-2.2/src/include -MT libublksrv_la-ublksrv.lo -MD -MP -MF .deps/libublksrv_la-ublksrv.Tpo -c ublksrv.c -o libublksrv_la-ublksrv.o >/dev/null 2>&1
mv -f .deps/libublksrv_la-ublksrv.Tpo .deps/libublksrv_la-ublksrv.Plo
/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -I../include -pthread -I/root/work/liburing-liburing-2.2/src/include -MT libublksrv_la-utils.lo -MD -MP -MF .deps/libublksrv_la-utils.Tpo -c -o libublksrv_la-utils.lo test -f 'utils.c' || echo './'utils.c
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -pthread -I/root/work/liburing-liburing-2.2/src/include -MT libublksrv_la-utils.lo -MD -MP -MF .deps/libublksrv_la-utils.Tpo -c utils.c -fPIC -DPIC -o .libs/libublksrv_la-utils.o
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -pthread -I/root/work/liburing-liburing-2.2/src/include -MT libublksrv_la-utils.lo -MD -MP -MF .deps/libublksrv_la-utils.Tpo -c utils.c -o libublksrv_la-utils.o >/dev/null 2>&1
mv -f .deps/libublksrv_la-utils.Tpo .deps/libublksrv_la-utils.Plo
/bin/sh ../libtool --tag=CXX --mode=link g++ -I/root/work/liburing-liburing-2.2/src/include -version-info 0:0:0 -L/root/work/liburing-liburing-2.2/src -o libublksrv.la -rpath /usr/local/lib libublksrv_la-ublksrv_cmd.lo libublksrv_la-ublksrv_json.lo libublksrv_la-ublksrv.lo libublksrv_la-utils.lo -L/usr/lib -luring
libtool: link: g++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-redhat-linux/10/../../../../lib64/crti.o /usr/lib/gcc/x86_64-redhat-linux/10/crtbeginS.o .libs/libublksrv_la-ublksrv_cmd.o .libs/libublksrv_la-ublksrv_json.o .libs/libublksrv_la-ublksrv.o .libs/libublksrv_la-utils.o -L/root/work/liburing-liburing-2.2/src -L/usr/lib -luring -L/usr/lib/gcc/x86_64-redhat-linux/10 -L/usr/lib/gcc/x86_64-redhat-linux/10/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/10/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-redhat-linux/10/crtendS.o /usr/lib/gcc/x86_64-redhat-linux/10/../../../../lib64/crtn.o -Wl,-soname -Wl,libublksrv.so.0 -o .libs/libublksrv.so.0.0.0
libtool: link: (cd ".libs" && rm -f "libublksrv.so.0" && ln -s "libublksrv.so.0.0.0" "libublksrv.so.0")
libtool: link: (cd ".libs" && rm -f "libublksrv.so" && ln -s "libublksrv.so.0.0.0" "libublksrv.so")
libtool: link: ar cru .libs/libublksrv.a libublksrv_la-ublksrv_cmd.o libublksrv_la-ublksrv_json.o libublksrv_la-ublksrv.o libublksrv_la-utils.o
libtool: link: ranlib .libs/libublksrv.a
libtool: link: ( cd ".libs" && rm -f "libublksrv.la" && ln -s "../libublksrv.la" "libublksrv.la" )
make[2]: Leaving directory '/root/work/ubdsrv/lib'
Making all in tests
make[2]: Entering directory '/root/work/ubdsrv/tests'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/root/work/ubdsrv/tests'
make[2]: Entering directory '/root/work/ubdsrv'
g++ -DHAVE_CONFIG_H -I. -I./include -fcoroutines -std=c++20 -I/root/work/liburing-liburing-2.2/src/include -MT ublk-ublksrv_tgt.o -MD -MP -MF .deps/ublk-ublksrv_tgt.Tpo -c -o ublk-ublksrv_tgt.o test -f 'ublksrv_tgt.cpp' || echo './'ublksrv_tgt.cpp
mv -f .deps/ublk-ublksrv_tgt.Tpo .deps/ublk-ublksrv_tgt.Po
g++ -DHAVE_CONFIG_H -I. -I./include -fcoroutines -std=c++20 -I/root/work/liburing-liburing-2.2/src/include -MT ublk-tgt_null.o -MD -MP -MF .deps/ublk-tgt_null.Tpo -c -o ublk-tgt_null.o test -f 'tgt_null.cpp' || echo './'tgt_null.cpp
mv -f .deps/ublk-tgt_null.Tpo .deps/ublk-tgt_null.Po
g++ -DHAVE_CONFIG_H -I. -I./include -fcoroutines -std=c++20 -I/root/work/liburing-liburing-2.2/src/include -MT ublk-tgt_loop.o -MD -MP -MF .deps/ublk-tgt_loop.Tpo -c -o ublk-tgt_loop.o test -f 'tgt_loop.cpp' || echo './'tgt_loop.cpp
mv -f .deps/ublk-tgt_loop.Tpo .deps/ublk-tgt_loop.Po
/bin/sh ./libtool --tag=CXX --mode=link g++ -fcoroutines -std=c++20 -I/root/work/liburing-liburing-2.2/src/include -L/root/work/liburing-liburing-2.2/src -o ublk ublk-ublksrv_tgt.o ublk-tgt_null.o ublk-tgt_loop.o lib/libublksrv.la -L/usr/lib -luring
libtool: link: g++ -fcoroutines -std=c++20 -I/root/work/liburing-liburing-2.2/src/include -o .libs/ublk ublk-ublksrv_tgt.o ublk-tgt_null.o ublk-tgt_loop.o -L/root/work/liburing-liburing-2.2/src lib/.libs/libublksrv.so -L/usr/lib -luring -Wl,-rpath -Wl,/usr/local/lib
/usr/bin/ld: ublk-ublksrv_tgt.o: in function ublksrv_drain_fetch_commands(ublksrv_dev*, ublksrv_queue_info*)': ublksrv_tgt.cpp:(.text+0x85e): undefined reference to pthread_join'
/usr/bin/ld: ublk-ublksrv_tgt.o: in function ublksrv_io_handler(void*)': ublksrv_tgt.cpp:(.text+0x9f8): undefined reference to pthread_create'
/usr/bin/ld: lib/.libs/libublksrv.so: undefined reference to `pthread_setaffinity_np'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:545: ublk] Error 1
make[2]: Leaving directory '/root/work/ubdsrv'
make[1]: *** [Makefile:693: all-recursive] Error 1
make[1]: Leaving directory '/root/work/ubdsrv'
make: *** [Makefile:431: all] Error 2

ublkdrv supporting zero copy

This might not be the right place to ask this question. But I saw you are working on some solution. @ming1
I am comparing two approaches of using a TCP NVMe-oF target from a host.

1. directly attach it to kernel
2. attach it to an SPDK application and expose it to kernel through ublkdrv
   When I run 4KB QD1 random write on them, the second approach shows an additional ~20us average latency and 50x tail latency (110 vs 5900 p99.99).
   I suspect most of the overhead comes from the memory copy and can be avoided with a zero-copy implementation.
   Do you have a working prototype of the zero-copy ublk driver I can try?

Documentation of the API functions

I am developing a userspace block device as an alternative solution to iSCSI block device in the Longhorn project. Based on preliminary performance evaluations, it appears that our userspace block device (longhorn-ublk) has the potential to improve the performance of Longhorn significantly.

It would be beneficial for the ubdsrv project to offer well-documented API documentation similar to http://libfuse.github.io/doxygen. This would provide users with a clear understanding of how to utilize the ubdsrv API, making it easier to develop customized block devices that integrate with it.

Thanks for the awesome project.

[build error]gettimeofday is undefined

eqcow2/qcow2.cpp: In constructor ‘MetaFlushingState::MetaFlushingState(Qcow2TopTable&, bool)’:
qcow2/qcow2.cpp:917:2: error: ‘gettimeofday’ was not declared in this scope; did you mean ‘SYS_gettimeofday’?
917 | gettimeofday(&last_flush, NULL);
| ^~~~~~~~~~~~
| SYS_gettimeofday
qcow2/qcow2.cpp: In member function ‘void MetaFlushingState::__done(Qcow2State&, ublksrv_queue*)’:
qcow2/qcow2.cpp:1121:2: error: ‘gettimeofday’ was not declared in this scope; did you mean ‘SYS_gettimeofday’?
1121 | gettimeofday(&last_flush, NULL);
| ^~~~~~~~~~~~
| SYS_gettimeofday
qcow2/qcow2.cpp: In member function ‘bool MetaFlushingState::__need_flush(int)’:
qcow2/qcow2.cpp:1324:3: error: ‘gettimeofday’ was not declared in this scope; did you mean ‘SYS_gettimeofday’?
1324 | gettimeofday(&t, NULL);
| ^~~~~~~~~~~~
| SYS_gettimeofday
make[2]: *** [Makefile:750: ublk-qcow2.o] Error 1
make[2]: Leaving directory '/root/work/ubdsrv'
make[1]: *** [Makefile:828: all-recursive] Error 1
make[1]: Leaving directory '/root/work/ubdsrv'
make: *** [Makefile:476: all] Error 2

Maybe we should include <sys/time.h> in qcow2/qcow2.h? I did so and it succeeded.

Cannot link to pthread

Hi, although #12 is helpful, I encountered the same problem again after I installed autoconf-archive. I used ./build_with_liburing_src to build ubdsrv. But the linker could not find pthread. See:

libtool: link: gcc -Wall -Werror -Wuninitialized -Wmaybe-uninitialized -Wno-sign-compare -Wno-parentheses -Wvla -Wframe-larger-than=5000 -Wstack-usage=10000 -pthread -I/root/liburing-liburing-2.2/src/include -g -O0 -o .libs/demo_event demo_event-demo_event.o -L/root/liburing-liburing-2.2/src lib/.libs/libublksrv.so -L/usr/lib -luring -pthread -Wl,-rpath -Wl,/usr/local/lib
mv -f qcow2/.deps/ublk-qcow2_flush_meta.Tpo qcow2/.deps/ublk-qcow2_flush_meta.Po
/bin/sh ./libtool --tag=CXX --mode=link g++ -fcoroutines -std=c++20 -I/root/liburing-liburing-2.2/src/include -g -O0 -L/root/liburing-liburing-2.2/src -o ublk ublk-ublksrv_tgt.o ublk-tgt_null.o ublk-tgt_loop.o qcow2/ublk-tgt_qcow2.o qcow2/ublk-qcow2.o qcow2/ublk-qcow2_meta.o qcow2/ublk-utils.o qcow2/ublk-qcow2_flush_meta.o lib/libublksrv.la -L/usr/lib -luring
libtool: link: g++ -fcoroutines -std=c++20 -I/root/liburing-liburing-2.2/src/include -g -O0 -o .libs/ublk ublk-ublksrv_tgt.o ublk-tgt_null.o ublk-tgt_loop.o qcow2/ublk-tgt_qcow2.o qcow2/ublk-qcow2.o qcow2/ublk-qcow2_meta.o qcow2/ublk-utils.o qcow2/ublk-qcow2_flush_meta.o -L/root/liburing-liburing-2.2/src lib/.libs/libublksrv.so -L/usr/lib -luring -Wl,-rpath -Wl,/usr/local/lib
/usr/bin/ld: ublk-ublksrv_tgt.o: in function ublksrv_drain_fetch_commands(ublksrv_dev*, ublksrv_queue_info*)': /root/ubdsrv/ublksrv_tgt.cpp:262: undefined reference to pthread_join'
/usr/bin/ld: ublk-ublksrv_tgt.o: in function ublksrv_io_handler(void*)': /root/ubdsrv/ublksrv_tgt.cpp:298: undefined reference to pthread_create'
/usr/bin/ld: lib/.libs/libublksrv.so: undefined reference to pthread_spin_init' /usr/bin/ld: lib/.libs/libublksrv.so: undefined reference to pthread_spin_unlock'
/usr/bin/ld: lib/.libs/libublksrv.so: undefined reference to pthread_spin_lock' /usr/bin/ld: lib/.libs/libublksrv.so: undefined reference to pthread_setaffinity_np'
collect2: error: ld returned 1 exit status

Looks like -lpthread is missing. So I edited build_with_liburing_src:

LDFLAGS="-L${LIBURING_DIR}/src -lpthread"

Then I ran build_with_liburing_src again. And the build succeeded.

runtime error need help when trying to build ublk server in ubuntu kernel 6.1.9

here is build error on ubuntu kernel 6.19. I also build and install latest liburing.

but when I try to add loop null, it shows below error, could you please shed light?

wayne@wayne-VirtualBox:~/wayne/ubdsrv$ ls
aclocal.m4               compile       config.log     configure.ac  demo_event.c             demo_null-demo_null.o  install-sh  ltmain.sh    Makefile.in  README.rst    tgt_null.cpp   ublksrv_tgt.cpp
autom4te.cache           config.guess  config.status  COPYING       demo_event-demo_event.o  depcomp                lib         m4           missing      stamp-h1      ublk           ublk-tgt_loop.o
build_with_liburing_src  config.h      config.sub     COPYING.LGPL  demo_null                doc                    libtool     Makefile     nbd          tests         ublksrv.pc     ublk-tgt_null.o
ci                       config.h.in   configure      demo_event    demo_null.c              include                LICENSE     Makefile.am  qcow2        tgt_loop.cpp  ublksrv.pc.in  ublk-ublksrv_tgt.o
wayne@wayne-VirtualBox:~/wayne/ubdsrv$ sudo ./ublk -h
unknown command: -h
ublk add -t {null|loop|qcow2|nbd} -n DEV_ID -q NR_HW_QUEUES -d QUEUE_DEPTH -u URING_COMP -g NEED_GET_DATA -r USER_RECOVERY -i USER_RECOVERY_REISSUE
           loop: -f backing_file [--buffered_io]
           	default is direct IO to backing file
           qcow2: -f backing_file
           nbd: --host=$HOST [--port=$PORT] | --unix=$UNIX_PATH
ublk del -n DEV_ID [-a | --all]
ublk list [-n DEV_ID]
wayne@wayne-VirtualBox:~/wayne/ubdsrv$ sudo ./ublk add -t null
can't get sqe ret -22
can't add dev -1, ret -22
wayne@wayne-VirtualBox:~/wayne/ubdsrv$ 

FR: Improve testing for users/builders

Currently tests always return success even on failures. For example if fio is not installed, exit code of all tests run is 0.

Also, it's hard to determine if some particular tests are failed, for example generic/001 output is not different if fio is installed or not, but perhaps it didn't work correctly without fio.

It would be good to have

1. Tests return failure if they fail
2. Define small subset tests that not taking much time for smoke testing after build.

An easier way to enable ublksrv log functions

Hi Ming,

I find that ublksrv provides log functions but it is disabled by default. Is there any easier ways to enable them?

We could add "-DDEBUG" in CFLAGS while running ./configure but I think this is not handy for users. Or we could add these steps in README.

#ifdef DEBUG
static inline void ublksrv_log(int priority, const char *fmt, ...)
{
    va_list ap;

    va_start(ap, fmt);
    vsyslog(priority, fmt, ap);
}

static inline void ublksrv_printf(FILE *stream, const char *fmt, ...)
{
    va_list ap;

    va_start(ap, fmt);
    vfprintf(stream, fmt, ap);
}
#else
static inline void ublksrv_log(int priority, const char *fmt, ...) { }
static inline void ublksrv_printf(FILE *stream, const char *fmt, ...) {}
#endif

Remove lib/ublksrv_aio.c

Hi Ming,

There is no user of lib/ublksrv_aio.c now except demo_event.c. What about removing it from lib/ so that the library is simpler? I think related code can be moved into demo_event.c.

BTW, lib/ublksrv_aio.c uses pthread_spin_lock(), eventfd()... which should not be included in a library.

ublk can't get sqe

I am trying to use ublk on an AWS EC2 instance.
The OS image I use is Ubuntu 22.04 LTS with the kernel version=6.7.6 (ubuntu image provided by AWS does not have ublk_drv module, so I updated the kernel version).

ubuntu@ip-172-31-86-113:~/ubdsrv$ uname -r
6.7.6-060706-generic

I installed liburing and ublksrv from source and did modprobe for ublk_drv.

However, it seems ublk cannot get sqe from the control ring.

ubuntu@ip-172-31-86-113:~/ubdsrv$ sudo ublk features
can't get sqe ret -22

Any ideas on what could possibly go wrong?

tgt_loop: hardcoded 512 logical block size causes zram unaligned I/O failures

Linux zram devices advertise a 4k logical block size by default:

# modprobe zram num_devices="1"
# echo "1G" > /sys/block/zram0/disksize
# cat /sys/devices/virtual/block/zram0/queue/logical_block_size 
4096

When using the tgt_loop backend to expose a 4k zram device via ublk, the logical block size is advertised as 512 bytes:

# ublk add -t loop -f /dev/zram0
# cat /sys/devices/virtual/block/ublkb0/queue/logical_block_size 
512

This results in 512 aligned I/O attempts from the block layer, instead of the 4k aligned request required by zram, causing I/O failures:

rapido1:/# mkdir /mnt && mkfs.xfs /dev/ublkb0 
meta-data=/dev/ublkb0            isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=0    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...Done.
rapido1:/# mount /dev/ublkb0 /mnt/
[   62.907137] I/O error, dev ublkb0, sector 0 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 2
[   62.908494] XFS (ublkb0): SB validate failed with error -5.
mount: /mnt: can't read superblock on /dev/ublkb0.

Changing the hardcoded struct ublk_params.basic.logical_bs_shift values to advertise 4k alignment avoids this error, e.g.:

--- a/tgt_loop.cpp
+++ b/tgt_loop.cpp
@@ -55,15 +55,15 @@ static int loop_init_tgt(struct ublksrv_dev *dev, int type, int argc, char
        struct ublk_params p = {
                .types = UBLK_PARAM_TYPE_BASIC | UBLK_PARAM_TYPE_DISCARD,
                .basic = {
-                       .logical_bs_shift       = 9,
+                       .logical_bs_shift       = 12,
                        .physical_bs_shift      = 12,
                        .io_opt_shift   = 12,
-                       .io_min_shift   = 9,
-                       .max_sectors            = info->max_io_buf_bytes >> 9,
+                       .io_min_shift   = 12,
+                       .max_sectors            = info->max_io_buf_bytes >> 12,
                },

It'd be nice if ublk tgt_loop obtained the I/O alignment requirements from the underlying loopback device and advertised the same values, or alternatively supported a blocksize parameter with ublk add.

BTW, thanks for working on ublk - it's an impressive feature :-)

ublk del --all always returns failure

I did simple smoke test for our CI and found that ublk del --all always exits with a failure (exitcode=237).

Looking at the code I think this is the casue:

cmd_dev_del iterates over MAX_NR_UBLK_DEVS number of devices:

        for (i = 0; i < MAX_NR_UBLK_DEVS; i++)
                ret = __cmd_dev_del(i, true);

1. This overrides exit status (of intermediate dels), and
2. Always reporting exit for last (128th) device (which is usually non-existent), which is -19 (that becomes exitcode 237).

Add tests cases in blktests

Hi Ming,

Shall we add some ublk tests in blktests? So it could be more convenient for testing. For example, our test group can easily run blktests and report test failure. But deploying ublksrv and built-in tests may be a little bit harder.

I think the work is simple because in blktests, there is a miniublk and we can move some useful tests in ublksrv into blktests. At least we can test ublk add/del in blktests I think.

Should we enable gnutls?

Is gnutls supposed to be enabled?

Currently nbd/nbd-client.c have:

#include "cliserv.h"

#if HAVE_GNUTLS && !defined(NOTLS)
#include "crypto-gnutls.h"
#endif

and nbd/cliserv.h have:

#define NOTLS

Is this intentional?

ublk-qcow2 work with kvm guests

just like https://static.sched.com/hosted_files/kvmforum2020/9c/KVMForum_2020_io_uring_passthrough_Stefano_Garzarella.pdf, make io_uring SQ/CQ ring share between ublk-qcow2 and virtio-blk driver, to make performace best

Can we let ublk support lower gcc version

Hi, Ming

Our users cannot compile ublksrv by simply typing "make" in the top directory since they only have gcc 8.5.0 which does not support c++ coroutine. Actually they just want to run demo_*.c which do not require any c++ coroutine.

It is hard to update GCC on some machine. So I think we should let user choose whether build with c++ coroutine files(maybe ublksrv_tgt c++ files) or not. If they have lower GCC version, they can build without ublksrv c++ files such as ublksrv_tgt.c, loop_tgt.c.