spdk/spdk

Storage Performance Development Kit

Storage Performance Development Kit

NOTE: The SPDK mailing list has moved to a new location. Please visit this URL to subscribe at the new location. Subscribers from the old location will not be automatically migrated to the new location.

The Storage Performance Development Kit (SPDK) provides a set of tools and libraries for writing high performance, scalable, user-mode storage applications. It achieves high performance by moving all of the necessary drivers into userspace and operating in a polled mode instead of relying on interrupts, which avoids kernel context switches and eliminates interrupt handling overhead.

The development kit currently includes:

• NVMe driver
• I/OAT (DMA engine) driver
• NVMe over Fabrics target
• iSCSI target
• vhost target
• Virtio-SCSI driver

In this readme

• Documentation
• Prerequisites
• Source Code
• Build
• Unit Tests
• Vagrant
• AWS
• Advanced Build Options
• Shared libraries
• Hugepages and Device Binding
• Example Code
• Contributing

Documentation

Doxygen API documentation is available, as well as a Porting Guide for porting SPDK to different frameworks and operating systems.

Source Code

git clone https://github.com/spdk/spdk
cd spdk
git submodule update --init

Prerequisites

The dependencies can be installed automatically by scripts/pkgdep.sh. The scripts/pkgdep.sh script will automatically install the bare minimum dependencies required to build SPDK. Use --help to see information on installing dependencies for optional components

./scripts/pkgdep.sh

Build

Linux:

./configure
make

FreeBSD: Note: Make sure you have the matching kernel source in /usr/src/ and also note that CONFIG_COVERAGE option is not available right now for FreeBSD builds.

./configure
gmake

Unit Tests

./test/unit/unittest.sh

You will see several error messages when running the unit tests, but they are part of the test suite. The final message at the end of the script indicates success or failure.

Vagrant

A Vagrant setup is also provided to create a Linux VM with a virtual NVMe controller to get up and running quickly. Currently this has been tested on MacOS, Ubuntu 16.04.2 LTS and Ubuntu 18.04.3 LTS with the VirtualBox and Libvirt provider. The VirtualBox Extension Pack or [Vagrant Libvirt] (https://github.com/vagrant-libvirt/vagrant-libvirt) must also be installed in order to get the required NVMe support.

Details on the Vagrant setup can be found in the SPDK Vagrant documentation.

AWS

The following setup is known to work on AWS: Image: Ubuntu 18.04 Before running setup.sh, run modprobe vfio-pci then: DRIVER_OVERRIDE=vfio-pci ./setup.sh

Advanced Build Options

Optional components and other build-time configuration are controlled by settings in the Makefile configuration file in the root of the repository. CONFIG contains the base settings for the configure script. This script generates a new file, mk/config.mk, that contains final build settings. For advanced configuration, there are a number of additional options to configure that may be used, or mk/config.mk can simply be created and edited by hand. A description of all possible options is located in CONFIG.

Boolean (on/off) options are configured with a 'y' (yes) or 'n' (no). For example, this line of CONFIG controls whether the optional RDMA (libibverbs) support is enabled:

CONFIG_RDMA?=n

To enable RDMA, this line may be added to mk/config.mk with a 'y' instead of 'n'. For the majority of options this can be done using the configure script. For example:

./configure --with-rdma

Additionally, CONFIG options may also be overridden on the make command line:

make CONFIG_RDMA=y

Users may wish to use a version of DPDK different from the submodule included in the SPDK repository. Note, this includes the ability to build not only from DPDK sources, but also just with the includes and libraries installed via the dpdk and dpdk-devel packages. To specify an alternate DPDK installation, run configure with the --with-dpdk option. For example:

Linux:

./configure --with-dpdk=/path/to/dpdk/x86_64-native-linuxapp-gcc
make

FreeBSD:

./configure --with-dpdk=/path/to/dpdk/x86_64-native-bsdapp-clang
gmake

The options specified on the make command line take precedence over the values in mk/config.mk. This can be useful if you, for example, generate a mk/config.mk using the configure script and then have one or two options (i.e. debug builds) that you wish to turn on and off frequently.

Shared libraries

By default, the build of the SPDK yields static libraries against which the SPDK applications and examples are linked. Configure option --with-shared provides the ability to produce SPDK shared libraries, in addition to the default static ones. Use of this flag also results in the SPDK executables linked to the shared versions of libraries. SPDK shared libraries by default, are located in ./build/lib. This includes the single SPDK shared lib encompassing all of the SPDK static libs (libspdk.so) as well as individual SPDK shared libs corresponding to each of the SPDK static ones.

In order to start a SPDK app linked with SPDK shared libraries, make sure to do the following steps:

• run ldconfig specifying the directory containing SPDK shared libraries
• provide proper LD_LIBRARY_PATH

If DPDK shared libraries are used, you may also need to add DPDK shared libraries to LD_LIBRARY_PATH

Linux:

./configure --with-shared
make
ldconfig -v -n ./build/lib
LD_LIBRARY_PATH=./build/lib/:./dpdk/build/lib/ ./build/bin/spdk_tgt

Hugepages and Device Binding

Before running an SPDK application, some hugepages must be allocated and any NVMe and I/OAT devices must be unbound from the native kernel drivers. SPDK includes a script to automate this process on both Linux and FreeBSD. This script should be run as root.

sudo scripts/setup.sh

Users may wish to configure a specific memory size. Below is an example of configuring 8192MB memory.

sudo HUGEMEM=8192 scripts/setup.sh

There are a lot of other environment variables that can be set to configure setup.sh for advanced users. To see the full list, run:

scripts/setup.sh --help

Target applications

After completing the build process, SPDK target applications can be found in spdk/build/bin directory:

• nvmf_tgt - SPDK NVMe over Fabrics target presents block devices over a fabrics,
• iscsi_tgt - SPDK iSCSI target runs I/O operations remotely with TCP/IP protocol,
• vhost - A vhost target provides a local storage service as a process running on a local machine,
• spdk_tgt - combines capabilities of all three applications.

SPDK runs in a polled mode, which means it continuously checks for operation completions. This approach assures faster response than interrupt mode, but also lessens usefulness of tools like top, which only shows 100% CPU usage for SPDK assigned cores. spdk_top is a program which simulates top application and uses SPDK's JSON RPC interface to present statistics about SPDK threads, pollers and CPU cores as an interactive list.

Example Code

Example code is located in the examples directory. The examples are compiled automatically as part of the build process. Simply call any of the examples with no arguments to see the help output. You'll likely need to run the examples as a privileged user (root) unless you've done additional configuration to grant your user permission to allocate huge pages and map devices through vfio.

Contributing

For additional details on how to get more involved in the community, including contributing code and participating in discussions and other activities, please refer to spdk.io

spdk's Issues

Segmentation fault (core dumped) after issuing write and trim in certain orders on same block.

I have written a little benchmark for testing delays for read, write and operations in different orders and different number of blocks, and it worked in an older version of spdk - I checked and that is 454 commits behind. With the current version I get a Segmentation fault (core dumped) whenever I do some certain permutations of commands.

These sequences work (r = read, w = write, t = trim): wrt, twr.
These fail: rwt, wtr, rtw.

Is this expected or is my code wrong?

Current algorithm (for rwt, it's similar for the other sequences, just change the order):

• Send read command

• Read completion callback issues write command.

• Write completion callback issues trim command.

• Trim completion does nothing (sets "done" flag to 1 so program ends).

  Should I remove the nesting? Issue commands outside the callback? This is a single thread only, just a sequence of three commands, reset everything, sleep half a second (was getting some noise), repeat to gather more timing data.

  Thank you.

elete

Abort with default configuration of dpdk2.1 (CONFIG_RTE_MAX_MEMSEG)

<...>
EAL: Requesting 512 pages of size 2MB from socket 0
EAL: Requesting 512 pages of size 2MB from socket 1
EAL: Can only reserve 569 pages from 1024 requested
Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
Please either increase it or request less amount of memory.

recompile dpdk with CONFIG_RTE_MAX_MEMSEG=512 works fine

Probe fails and device does not show up anymore, even after reboot, after resetting to nvme.

Clean DPDK/SPDK checkouts.
Ran setup.sh script.
Ran hello_world example and this is the output, returned instantly.

EAL: Detected 16 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:82:00.0 on NUMA socket 1
EAL:   probe driver: 1b85:6018 spdk_nvme
Initializing NVMe Controllers
EAL: PCI device 0000:82:00.0 on NUMA socket 1
EAL:   probe driver: 1b85:6018 spdk_nvme
Attaching to 0000:82:00.0
nvme_ctrlr.c:1096:nvme_ctrlr_process_init: ***ERROR*** get registers failed
spdk_nvme_probe() failed

Ran it again and it hangs:

...
Attaching to 0000:82:00.0
Attached to 0000:82:00.0
Using controller TOSHIBA-RD400        (        X63S1007TPEV) with 1 namespaces.
  Namespace ID: 1 size: 256GB
Initialization complete.
^C

Third time:

..
EAL: PCI device 0000:82:00.0 on NUMA socket 1
EAL:   probe driver: 1b85:6018 spdk_nvme
Attaching to 0000:82:00.0
nvme_ctrlr.c:1211:nvme_ctrlr_process_init: ***ERROR*** Initialization timed out in state 2
nvme_ctrlr.c: 403:nvme_ctrlr_shutdown: ***ERROR*** did not shutdown within 5 seconds
spdk_nvme_probe() failed

At this point SPDK doesn't work, device is still listed in /dev as uio0.
So I tried resetting to nvme, ran the script and the device is gone, no nvme* nor uio in /dev.

Rebooted the machine, device is still gone, no nvme in /dev and it doesn't even show in lspci anymore. We found two solutions for this: turn off the machine and switch PCI ports or cut the power of the machine for a few seconds and power it back on.

Kernel version:
Linux version 4.8.12 (***) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.2) ) #20 SMP Mon Jan 30 19:40:03 CST 2017

What is this? Is it a kernel or SPDK issue?

Right now we are considering moving to older versions, 16.06 segfaults but 16.08 seems to work (at least the examples), it just sucks that a lot of code needs to be modified all over again and we will need to rollback DPDK also.

Thank you.

Is there a roadmap on porting this project to Mac OS X? BSD's ?

Should it be assumed that this will work out of the box.

NVME device insertion/removal?

I don't see anything in the SPDK regarding device removal or insertion.
Is this going to be added at some point?
I know a lot of the current work was done with Add In Cards but how will this work with SFF 8639 devices?
I also see that there are is no validation of register accesses.
Blindly assuming PCIe accesses are successful can lead to data corruption.
Isn't it a rule that PCIe register accesses should always check the result for -1 (0xffffffff indicates an access failure).

thank you

chuck

Should ctrlr_lock be acquired when an admin command is submitted?

Some functions get ctrlr_lock when they call nvme_ctrlr_submit_admin_request, but some functions don't.
Should we get ctrlr_lock in nvme_ctrlr_submit_admin_request function and change all caller functions not to get lock?

Ex;
functions that get lock
spdk_nvme_ctrlr_cmd_admin_raw, nvme_ctrlr_cmd_attach_ns, etc

functions that don't get lock
nvme_ctrlr_cmd_identify_controller, nvme_ctrlr_cmd_identify_namespace, etc

NVMF target creation failure

After the recent changes in spdk I got this error while creating the NVMF target

app/nvmf_tgt/nvmf_tgt -c etc/spdk/nvmf.conf.in
Starting Intel(R) DPDK initialization ...
[ DPDK EAL parameters: nvmf -c 1 -n 4 -m 2048 --master-lcore=0 --file-prefix=rte0 --proc-type=auto ]
EAL: Detected 32 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 1c58:3 spdk_nvme
EAL: PCI device 0000:82:00.0 on NUMA socket 1
EAL: probe driver: 1c58:3 spdk_nvme
done.
Occupied cpu core mask is 0x1
Occupied cpu socket mask is 0x1
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 1c58:3 spdk_nvme
EAL: PCI device 0000:82:00.0 on NUMA socket 1
EAL: probe driver: 1c58:3 spdk_nvme
Ioat Copy Engine Offload Enabled
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 1c58:3 spdk_nvme
Probing device 0000:81:00.0
EAL: PCI device 0000:82:00.0 on NUMA socket 1
EAL: probe driver: 1c58:3 spdk_nvme
Probing device 0000:82:00.0
Total cores available: 1
Reactor started on core 0 on socket 0
*** RDMA Transport Init ***
allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 0 on socket 0
allocated subsystem nqn.2016-06.io.spdk:cnode11_1 on lcore 0 on socket 0
*** NVMf Target Listening on 10.30.0.11 port 4420 ***
conf.c: 573:spdk_nvmf_parse_subsystem: ERROR Could not find NVMe controller for Subsystem1
nvmf_tgt.c: 336:spdk_nvmf_startup: ERROR spdk_nvmf_parse_conf() failed

Here is my etc/spdk/nvmf.conf.in

[Global]

LogFacility "local7"

[Rpc]
Enable No
Listen 127.0.0.1

[Malloc]
NumberOfLuns 8
LunSizeInMB 64

[Nvmf]
MaxQueuesPerSession 16

MaxQueueDepth 128

AcceptorCore 8

AcceptorPollRate 10000

[Nvme]
ResetControllerOnTimeout Yes
NvmeTimeoutValue 30
AdminPollRate 100000

[Subsystem1]
NQN nqn.2016-06.io.spdk:cnode11_1
Core 9
Mode Direct
Listen RDMA 10.30.0.11:4420
NVMe 0000:81:00.0

[Subsystem2]
NQN nqn.2016-06.io.spdk:cnode11_2
Core 10
Mode Direct
Listen RDMA 10.20.0.11:4420
NVMe 0000:82:00.0

All the hello_world and perf and other examples works fine.

Full stack hotplug support was dropped?

I noticed that full stack hotplug support is removed from the following release note.

https://github.com/spdk/spdk/wiki/Release-17.03

Will SPDK support it in the next release? I really expect it.

spdk_zmalloc failed

I was trying to evaluate my own NVMe SSD with perf tool executed with block size set to 64KB (e.g., perf -q 1 -s 65536 -w read -t 5). Unfortunately, the test has been failed with the spdk_zmalloc failed. Before this test, 1GB hugepages was allocated. Could let me know how to resolve this issue?

Should spdk_bdev_io be returned to the caller in spdk_bdev_cleanup_pending_rbuf_io()?

spdk_bdev_cleanup_pending_rbuf_io function removes spdk_bdev_io structures from g_need_rbuf_small list or g_need_rbuf_large list. Then anybody doesn't care about them and they don't go back to spdk_bdev_g_io_pool. I think these spdk_bdev_io should be returned to the caller.

Support For Big Endian Systems

Hi,

I can't find in code where do we format the cq_head/sq_tail to little endian format before writing it to doorbell. isn't it a necessary step?

Thanks,
Roy

librte_malloc has been moved to eal/common in DPDK. Seems no librte_malloc anymore.

I am using the current version of DPDK from github.
Since there is no librte_malloc, in mk/spdk.common.mk, -lrte_malloc needs to be removed from DPDK_LIB for new versions of dpdk.

spdk_nvmf_ctrlr_create()?

Where does the function be defined?
I don't find it!

Minimum hardware requirement

... to start testing/integrating/playing with SPDK ? AFAIK, for DPDK I need an Intel based NIC - which I already have in our Dell R610 servers - but how about NVMe/SSD ? Any decently priced card I could buy to install in a Dell R610 ? Thank you in advance for your answer.

When does lun->ref have more than 1?

I think spdk_scsi_lun_delete() should call spdk_scsi_lun_free_io_channel(). spdk_scsi_lun_free_io_channel() need to be called lun->ref times to free the io_channel. But I don't find the case where lun->ref becomes 2 or bigger.

make fails: pci.c:47:23: fatal error: pciaccess.h: No such file or directory

Downloaded the latest spdk package (spdk-16.08). make fails trying to find <pciaccess.h>. Please help.

#> make DPDK_DIR=./dpdk-16.07/x86_64-native-linuxapp-gcc

CC lib/util/pci.o
pci.c:47:23: fatal error: pciaccess.h: No such file or directory
#include <pciaccess.h>
^
compilation terminated.

Running as a non-privileged user with vfio

Investigate running as a non-privileged user with vfio. This requires VT-d and the IOMMU enabled on a system. The expectation is that this just works and only needs to be documented, but it may require a few script updates. Running with vfio as root already works.

If anyone from the wider community would like to take a look at this, we'd happily accept any necessary patches.

Display more useful Ceph information for command ‘lsscsi’.

when input "lsscsi", the rbd device info will be displayed as "/dev/sdX Ceph rbd", and it is hard coded. We need more useful information, such as pool name and image name.

Should NVMe PMD driver check CSTS.CFS field in nvme_pcie_qpair_complete_tracker()?

The following sentences are in NVMe specification 1.2.1 in page 217.

9.5 Controller Fatal Status Condition
If the controller has a serious error condition and is unable to communicate with host software via completion queue entries in the Admin or I/O Completion Queues, then the controller may set the Controller Fatal Status (CSTS.CFS) field to ‘1’. This indicates to host software that a serious error condition has occurred. When this condition occurs, host software should reset and then re-initialize the controller.
The Controller Fatal Status condition is not indicated with an interrupt. If host software experiences timeout conditions and/or repeated errors, then host software should consult the Controller Fatal Status (CSTS.CFS) field to determine if a more serious error has occurred.

Should NVMe PMD driver check CSTS.CFS in nvme_pcie_qpair_complete_tracker() and reset controller if CSTS.CFS is 1?

Abort of requests that are on the queuedReq list

In the case that an SQ is full the code queues new requests to:
STAILQ_HEAD(, nvmeRequest) queuedReq; of the corresponding Queue Pair Struct.
However, when nvme_ctrlr_cmd_abort is called it issues an NVMe Abort command to the corresponding Controllers Admin SQ.
Shouldn't the code first search the queuedReq list for the target request?
Also, if the Request to be aborted is on the queuedReq list the cid will be 0 as it hasn't been assigned a Tracker yet. The way the code is today this would result in an Abort being issued for the wrong Request, meaning the Request with cid = 0 will be aborted by the drive.

thank you

chuck

Repeatable Defect: blk_update_request: I/O errors; lost async page write; nvme0: reconnecting in 10 seconds

Hello,

We have been experimenting with SPDK and the Linux based kernel initiator. We have ran into a repeatable issue regarding the usage of non-direct I/O directly to a NVMeF block device. We also ran into similar issues when using a file system with direct I/O turned off. In any case lets focus on the case of I/Os to a raw block device as it seems easier to replicate the bug.

I have been unable to diagnose the root cause of the problem to fix it so I am posting here to hopefully get some help.

To replicate:

1. Get latest SPDK/DPDK sources and compile on target node with NVMe drives (note: we have two separate RNICs on the target, but have configured the SPDK configuration file to place the NVMe drives on the correct root complex for each RNIC - it is a two socket system... I don't think the RNICs are the issue just wanted to point that detail out in our configuration). Also we are using IB as the transport with Mellanox Connect-IB cards.

2. Do discovery and connect via the regular 'nvme' open source initiator package. Note: no file system involved at all with this test.

3. The following FIO command does not cause the bug:

fio --rw=randwrite --bs=128k --numjobs=1 --iodepth=256 --loops=1 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --exitall --name test --filename=/dev/nvme0n1 --output=blah --time_based --runtime=60 --group_reporting --size=30G

4. The following FIO command does cause the bug:

fio --rw=randwrite --bs=128k --numjobs=1 --iodepth=256 --loops=1 --ioengine=libaio --direct=0 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --exitall --name test --filename=/dev/nvme0n1 --output=blah --time_based --runtime=60 --group_reporting --size=30G

Note: the only difference in the two commands is the --direct flag change.

5. If you run FIO command from step 4 you will see on the initiator in the dmesg things like the following:

[158052.975693] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.4.74:4420
[158058.986892] nvme nvme0: creating 56 I/O queues.
[158061.990454] nvme nvme0: new ctrl: NQN "nqn.2016-06.io.spdk:cnode8", addr 192.168.4.74:4420
[158697.246241] blk_update_request: I/O error, dev nvme0n1, sector 207104
[158697.248259] Buffer I/O error on dev nvme0n1, logical block 25888, lost async page write
[158697.250775] Buffer I/O error on dev nvme0n1, logical block 25889, lost async page write
[158697.253274] Buffer I/O error on dev nvme0n1, logical block 25890, lost async page write
[158697.255779] Buffer I/O error on dev nvme0n1, logical block 25891, lost async page write

...

[158697.273322] blk_update_request: I/O error, dev nvme0n1, sector 207616
[158697.275357] blk_update_request: I/O error, dev nvme0n1, sector 208128
[158697.277395] blk_update_request: I/O error, dev nvme0n1, sector 208384
[158697.279431] blk_update_request: I/O error, dev nvme0n1, sector 209408
[158697.281473] blk_update_request: I/O error, dev nvme0n1, sector 209920

...

[158697.473874] nvme nvme0: reconnecting in 10 seconds
[158709.884153] nvme nvme0: Successfully reconnected
[158727.572304] blk_update_request: 29 callbacks suppressed
[158727.572311] blk_update_request: I/O error, dev nvme0n1, sector 228352
[158727.572388] nvme nvme0: reconnecting in 10 seconds
[158727.614845] buffer_io_error: 1238 callbacks suppressed
[158727.614849] Buffer I/O error on dev nvme0n1, logical block 28544, lost async page write
[158727.698281] Buffer I/O error on dev nvme0n1, logical block 28545, lost async page write

...

[158738.976347] nvme nvme0: Successfully reconnected

Please let me know if you require more information about how we have setup the SPDK configuration file as well as any other details related to our setup if you have issues replicating this bug.

Appreciate any help/advice you all can offer.

Thanks.
-Matt

perf: INTERNAL DEVICE ERROR (00/06) nvme_set_num_queues failed!

$ sudo ./perf -s 4096 -q 64 -w randrw -M 0 -c 1 -t 60
...
Created task_pool
Initializing NVMe Controllers
SET FEATURES (09) sqid:0 cid:15 nsid:0 cdw10:00000007 cdw11:ffffffff
INTERNAL DEVICE ERROR (00/06) sqid:0 cid:15 cdw0:0 sqhd:0002 p:1 m:0 dnr:0
nvme_set_num_queues failed!
nvme_attach failed for controller at pci bdf 4:0:0

$ sudo ./identify
...
EAL: TSC frequency is ~2299998 KHz
EAL: Master lcore 0 is ready (tid=feab4940;cpuset=[0])
SET FEATURES (09) sqid:0 cid:15 nsid:0 cdw10:00000007 cdw11:ffffffff
INTERNAL DEVICE ERROR (00/06) sqid:0 cid:15 cdw0:0 sqhd:0002 p:1 m:0 dnr:0
nvme_set_num_queues failed!
failed to attach to NVMe controller at PCI BDF 4:0:0

$ sudo ./nvme-cli/nvme error-log /dev/nvme0
NVMe Status:INTERNAL(6)

When will NvmeLunsPerNs is removed?

When nvme bdev is deleted by rpc.py with delete_lun option, coresponding nvme_device is not deleted.

ex: Nvme3n1p0 is on 0000:02:00.0
# ./scripts/rpc.py delete_lun Nvme3n1p0

Then I try to create bdev again by rpc.py with construct_nvme_bdev. But it fails because the nvme_device already existed.

# ./scripts/rpc.py construct_nvme_bdev 0000:02:00.0
Got JSON-RPC error response
request:
{
  "jsonrpc": "2.0",
  "params": {
    "pci_address": "0000:02:00.0"
  },
  "method": "construct_nvme_bdev",
  "id": 1
}
response:
{
  "message": "Invalid parameters",
  "code": -32602
}

I'd like to make a PR to delete nvme_device in blockdev_nvme_destruct.
Regarding to it, I have a question. Daniel mentioned that NvmeLunsPerNs option will be removed in #85. I want to wait for it, then make a PR. Would you tell me when NvmeLunsPerNs will be removed?

NVMF target perf test

I'm trying to test nvmf target using spdk/examples/perf/perf utility. I have 2 nodes, first one runs target with this config

[Global]
ReactorMask 0xFFFFFFFF
LogFacility "local7"

[Rpc
Enable No
Listen 127.0.0.1

[Nvmf]
MaxQueuesPerSession 16
MaxQueueDepth 128
AcceptorCore 8
AcceptorPollRate 10000

[Subsystem1]
NQN nqn.2016-06.io.spdk:cnode11_1
Core 9
Mode Direct
Listen RDMA 10.30.0.11:4420
NVMe 0000:81:00.0

[Subsystem2]
NQN nqn.2016-06.io.spdk:cnode11_2
Core 10
Mode Direct
Listen RDMA 10.20.0.11:4420
NVMe 0000:82:00.0

On the second node I run
spdk/examples/nvme/perf/perf -w randread -q 32 -s 4096 -t 5 -r 'trtype:RDMA adrfam:IPv4 traddr:10.30.0.11 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode11_1'
EAL: Detected 32 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 1c58:3 spdk_nvme
EAL: PCI device 0000:82:00.0 on NUMA socket 1
EAL: probe driver: 1c58:3 spdk_nvme
Initializing NVMe Controllers
Attaching to NVMe over Fabrics controller at 10.30.0.11:4420: nqn.2016-06.io.spdk:cnode11_1
nvme_ctrlr.c: 849:nvme_ctrlr_configure_aer: ERROR nvme_ctrlr_cmd_set_async_event_config failed!
Attached to NVMe over Fabrics controller at 10.30.0.11:4420: nqn.2016-06.io.spdk:cnode11_1
Associating HUSPR3216AHP301 (STM0001B5503 ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0

Device Information : IOPS MB/s Average min max
HUSPR3216AHP301 (STM0001B5503 ) from core 0: 231593.20 904.66 138.12 16.68 1254.84
Total : 231593.20 904.66 138.12 16.68 1254.84

I am just curious whether this Error is important.
The second question is should I try connecting to target nqn directly or use discovery service nqn or no nqn at all?

Invalid read when connecting with iscsi-perf twice while using an AIO device

Description of the problem:
Trying to do random reads with two different libiscsi based connections (same initiator) to a 128MByte block device via the the SPDK iSCSI target eventually results in a segfault due to an invalid memory read.

Steps to reproduce:

1. Create a 128MByte block device by doing:

echo "0 262144 zero" | dmsetup create fake

2. Add the something like the following to the iscsi.conf file:

[AIO]
  AIO /dev/mapper/fake

[TargetNode1]
  TargetName disk1
  TargetAlias "Data Disk1"
  Mapping PortalGroup1 InitiatorGroup1
  UseDigest Auto
  LUN1 AIO0
  QueueDepth 128

3. Run
   app/iscsi_tgt/iscsi_tgt -c iscsi.conf
4. In another terminal run iscsi-perf (from libiscsi) by doing

iscsi-perf -b 1 -r -m 1 iscsi://127.0.0.1/iqn.2016-06.io.spdk:disk1/1

5. In another terminal run iscsi-perf by doing

iscsi-perf -b 1 -r -m 1 iscsi://127.0.0.1/iqn.2016-06.io.spdk:disk1/1

Expected result:
Random I/O to do be done.

Actual result:
After about two minutes a segfault occurs.

How reproducible is the issue?
Happens every time (if you wait long enough).

Additional information:
Valgrind prints this:

Login from iqn.2010-11.libiscsi:iscsi-perf (127.0.0.1) on iqn.2016-06.io.spdk:disk1 tgt_node1 (127.0.0.1:3260,1), ISID=160b73ec8, TSIH=1, CID=0, HeaderDigest=off, DataDigest=off
Launching connection on acceptor thread
Login from iqn.2010-11.libiscsi:iscsi-perf (127.0.0.1) on iqn.2016-06.io.spdk:disk1 tgt_node1 (127.0.0.1:3260,1), ISID=160b73b08, TSIH=2, CID=0, HeaderDigest=off, DataDigest=off
==21377== Invalid read of size 8
==21377==    at 0x40E9D4: blockdev_aio_readv (blockdev_aio.c:117)
==21377==    by 0x40E9D4: blockdev_aio_get_rbuf_cb (blockdev_aio.c:236)
==21377==    by 0x42C497: spdk_bdev_io_set_rbuf (bdev.c:122)
==21377==    by 0x42C497: _spdk_bdev_io_get_rbuf (bdev.c:387)
==21377==    by 0x42C497: spdk_bdev_io_get_rbuf (bdev.c:893)
==21377==    by 0x40EAC0: _blockdev_aio_submit_request (blockdev_aio.c:249)
==21377==    by 0x40EAC0: blockdev_aio_submit_request (blockdev_aio.c:279)
==21377==    by 0x42B8A7: __submit_request (bdev.c:421)
==21377==    by 0x42B8A7: spdk_bdev_io_submit (bdev.c:457)
==21377==    by 0x42BC29: spdk_bdev_readv (bdev.c:618)
==21377==    by 0x4255D6: spdk_bdev_scsi_read (scsi_bdev.c:1300)
==21377==    by 0x4255D6: spdk_bdev_scsi_readwrite (scsi_bdev.c:1425)
==21377==    by 0x4259A9: spdk_bdev_scsi_process_block (scsi_bdev.c:1562)
==21377==    by 0x4259A9: spdk_bdev_scsi_execute (scsi_bdev.c:1903)
==21377==    by 0x424158: spdk_scsi_lun_execute_tasks (lun.c:242)
==21377==    by 0x418F4A: spdk_iscsi_op_data (iscsi.c:4102)
==21377==    by 0x418F4A: spdk_iscsi_execute (iscsi.c:4367)
==21377==    by 0x411E6D: spdk_iscsi_conn_handle_incoming_pdus (conn.c:1209)
==21377==    by 0x411E6D: spdk_iscsi_conn_execute (conn.c:1257)
==21377==    by 0x412160: spdk_iscsi_conn_full_feature_do_work (conn.c:1336)
==21377==    by 0x432725: _spdk_reactor_run (reactor.c:328)
==21377==  Address 0x30 is not stack'd, malloc'd or (recently) free'd

This issue doesn't seem to happen when using memory/malloc based targets.

A similar looking valgrind backtrace can be generated by using

iscsi-perf -b 32 -r -m 64 iscsi://127.0.0.1/iqn.2016-06.io.spdk:disk1/1

in two different terminals and then pressing Ctrl-C on one the iscsi-perf tasks.

Version information:
spdk commit b8da09b
dpdk-16.11
libiscsi 3a598ea8465c8204688c6ccbc6d74003342f6651

seg fault in vtophys_get_map vtophys.c:102

Program received signal SIGSEGV, Segmentation fault.
vtophys_get_map (vfn_2mb=8796093022115) at vtophys.c:102
102 map_1gb = vtophys_map_128tb.map[idx_128tb];
(gdb) bt
#0 vtophys_get_map (vfn_2mb=8796093022115) at vtophys.c:102
#1 spdk_vtophys (buf=buf@entry=0xfffffffff4799000) at vtophys.c:168
#2 0x00007ffff79757f7 in _nvme_qpair_build_contig_request (tr=0x7ffff5774000, req=0x7fffd91dd780, qpair=0x60a040)

at nvme_qpair.c:639

#3 nvme_qpair_submit_request (qpair=qpair@entry=0x60a040, req=0x7fffd91dd780) at nvme_qpair.c:903
#4 0x00007ffff79743fb in spdk_nvme_ns_cmd_read (ns=, qpair=0x60a040, buffer=,

lba=<optimized out>, lba_count=<optimized out>, cb_fn=<optimized out>, cb_arg=0x7ffff5dee000, io_flags=0)
at nvme_ns_cmd.c:238

buffer was allocated as

buf = rte_malloc(NULL, 4096, 4096);

One queue per thread limitation

The spdk does not expose queues to the user. Instead it assignes one queue per thread. Now, if I want to handle 1000 concurrent streams I can either spawn 1000 threads or use number of core threads (say 32) and do stream to nvme queue multiplexing in software. How does spdk map the 65K virtual queues to the (32) hardware queues? Is there any advantage in performance in letting the spdk/nvme doing the multiplexing over doing it in software? Running too many threads has its own issues so I would like to limit their number to the amount of cores.

SPDK on Samsung NVMe a802 Requested device 0000:05:00.0 cannot be used

Hello,

I am trying to run SPDK on Samsung NVMe. After the boot, the /dev/nvme devices show up. Running the scripts/setup.sh script renames the device to /dev/vfio/vfio. When running an example, I get the following error message. Can you suggest what might be the problem. I tried disabling intel_iommu off and on.

$ sudo examples/nvme/hello_world/hello_world

EAL: Detected 16 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL: probe driver: 144d:a802 spdk_nvme
EAL: Error - exiting with code: 1
Cause: Requested device 0000:05:00.0 cannot be used

Bad aio read when using the iSCSI target with a large block device?

Description of the problem:
Trying to do random access on a 2TByte block device based disk via the the SPDK iSCSI target results in a segfault due to an invalid memory read.

Steps to reproduce:

1. Create a 2TByte block device by doing:

echo "0 4294967296 zero" | dmsetup create fake

2. Add the something like the following to the iscsi.conf file:

[AIO]
  AIO /dev/mapper/fake

[TargetNode1]
  TargetName disk1
  TargetAlias "Data Disk1"
  Mapping PortalGroup1 InitiatorGroup1
  UseDigest Auto
  LUN1 AIO0
  QueueDepth 128

3. Run
   app/iscsi_tgt/iscsi_tgt -c iscsi.conf
4. In another terminal run iscsi-perf from libiscsi by doing

iscsi-perf -b 1 -r -m 1 iscsi://127.0.0.1/iqn.2016-06.io.spdk:disk1/1

Expected result:
Random I/O to do be done.

Actual result:
No I/O seems to take place but the following is printed by the SPDK iSCSI target:

Login from iqn.2010-11.libiscsi:iscsi-perf (127.0.0.1) on iqn.2016-06.io.spdk:disk1 tgt_node1 (127.0.0.1:3260,1), ISID=7fbe6b573ec8, TSIH=1, CID=0, HeaderDigest=off, DataDigest=off
conn.c: 844:spdk_iscsi_conn_read_data: ***ERROR*** Socket read error(104): Connection reset by peer
writev: Broken pipe
Launching connection on acceptor thread
Login from iqn.2010-11.libiscsi:iscsi-perf (127.0.0.1) on iqn.2016-06.io.spdk:disk1 tgt_node1 (127.0.0.1:3260,1), ISID=7fbe6b56fa08, TSIH=2, CID=0, HeaderDigest=off, DataDigest=off
iscsi_subsystem.c: 493:spdk_put_pdu: ***ERROR*** Negative PDU refcount: 0x7fbe6b573d80
Terminating connections(tsih 1): 0

After a short wait the following is printed again and again:

iscsi_subsystem.c: 493:spdk_put_pdu: ***ERROR*** Negative PDU refcount: 0x7fbe6b573d80
iscsi_subsystem.c: 493:spdk_put_pdu: ***ERROR*** Negative PDU refcount: 0x7fbe6b573d80

How reproducible is the issue?
Happens every time.

Additional information:
Running under valgrind prints this:

conn.c: 844:spdk_iscsi_conn_read_data: ***ERROR*** Socket read error(104): Connection reset by peer
writev: Broken pipe
Login from iqn.2010-11.libiscsi:iscsi-perf (127.0.0.1) on iqn.2016-06.io.spdk:disk1 tgt_node1 (127.0.0.1:3260,1), ISID=160b71808, TSIH=2, CID=0, HeaderDigest=off, DataDigest=off
iscsi_subsystem.c: 493:spdk_put_pdu: ***ERROR*** Negative PDU refcount: 0x160b73d80
Terminating connections(tsih 1): 0
==18350== Invalid read of size 8
==18350==    at 0x40E9D4: blockdev_aio_readv (blockdev_aio.c:117)
==18350==    by 0x40E9D4: blockdev_aio_get_rbuf_cb (blockdev_aio.c:236)
==18350==    by 0x42C497: spdk_bdev_io_set_rbuf (bdev.c:122)
==18350==    by 0x42C497: _spdk_bdev_io_get_rbuf (bdev.c:387)
==18350==    by 0x42C497: spdk_bdev_io_get_rbuf (bdev.c:893)
==18350==    by 0x40EAC0: _blockdev_aio_submit_request (blockdev_aio.c:249)
==18350==    by 0x40EAC0: blockdev_aio_submit_request (blockdev_aio.c:279)
==18350==    by 0x42B8A7: __submit_request (bdev.c:421)
==18350==    by 0x42B8A7: spdk_bdev_io_submit (bdev.c:457)
==18350==    by 0x42BC29: spdk_bdev_readv (bdev.c:618)
==18350==    by 0x4255D6: spdk_bdev_scsi_read (scsi_bdev.c:1300)
==18350==    by 0x4255D6: spdk_bdev_scsi_readwrite (scsi_bdev.c:1425)
==18350==    by 0x4259A9: spdk_bdev_scsi_process_block (scsi_bdev.c:1562)
==18350==    by 0x4259A9: spdk_bdev_scsi_execute (scsi_bdev.c:1903)
==18350==    by 0x424158: spdk_scsi_lun_execute_tasks (lun.c:242)
==18350==    by 0x414F7C: spdk_iscsi_conn_handle_queued_tasks (iscsi.c:2838)
==18350==    by 0x419CBB: spdk_iscsi_op_scsi_read (iscsi.c:2884)
==18350==    by 0x419CBB: spdk_iscsi_op_scsi (iscsi.c:2947)
==18350==    by 0x419CBB: spdk_iscsi_execute (iscsi.c:4336)
==18350==    by 0x411E6D: spdk_iscsi_conn_handle_incoming_pdus (conn.c:1209)
==18350==    by 0x411E6D: spdk_iscsi_conn_execute (conn.c:1257)
==18350==    by 0x412160: spdk_iscsi_conn_full_feature_do_work (conn.c:1336)
==18350==  Address 0x30 is not stack'd, malloc'd or (recently) free'd
==18350== 
==18350== 
==18350== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==18350==  Access not within mapped region at address 0x30
==18350==    at 0x40E9D4: blockdev_aio_readv (blockdev_aio.c:117)
==18350==    by 0x40E9D4: blockdev_aio_get_rbuf_cb (blockdev_aio.c:236)
==18350==    by 0x42C497: spdk_bdev_io_set_rbuf (bdev.c:122)
==18350==    by 0x42C497: _spdk_bdev_io_get_rbuf (bdev.c:387)
==18350==    by 0x42C497: spdk_bdev_io_get_rbuf (bdev.c:893)
==18350==    by 0x40EAC0: _blockdev_aio_submit_request (blockdev_aio.c:249)
==18350==    by 0x40EAC0: blockdev_aio_submit_request (blockdev_aio.c:279)
==18350==    by 0x42B8A7: __submit_request (bdev.c:421)
==18350==    by 0x42B8A7: spdk_bdev_io_submit (bdev.c:457)
==18350==    by 0x42BC29: spdk_bdev_readv (bdev.c:618)
==18350==    by 0x4255D6: spdk_bdev_scsi_read (scsi_bdev.c:1300)
==18350==    by 0x4255D6: spdk_bdev_scsi_readwrite (scsi_bdev.c:1425)
==18350==    by 0x4259A9: spdk_bdev_scsi_process_block (scsi_bdev.c:1562)
==18350==    by 0x4259A9: spdk_bdev_scsi_execute (scsi_bdev.c:1903)
==18350==    by 0x424158: spdk_scsi_lun_execute_tasks (lun.c:242)
==18350==    by 0x414F7C: spdk_iscsi_conn_handle_queued_tasks (iscsi.c:2838)
==18350==    by 0x419CBB: spdk_iscsi_op_scsi_read (iscsi.c:2884)
==18350==    by 0x419CBB: spdk_iscsi_op_scsi (iscsi.c:2947)
==18350==    by 0x419CBB: spdk_iscsi_execute (iscsi.c:4336)
==18350==    by 0x411E6D: spdk_iscsi_conn_handle_incoming_pdus (conn.c:1209)
==18350==    by 0x411E6D: spdk_iscsi_conn_execute (conn.c:1257)
==18350==    by 0x412160: spdk_iscsi_conn_full_feature_do_work (conn.c:1336)
==18350==  If you believe this happened as a result of a stack
==18350==  overflow in your program's main thread (unlikely but
==18350==  possible), you can try to increase the size of the
==18350==  main thread stack using the --main-stacksize= flag.
==18350==  The main thread stack size used in this run was 8388608.

Version information:
spdk commit b8da09b
dpdk-16.11
libiscsi 3a598ea8465c8204688c6ccbc6d74003342f6651

SPDK can't attach nvme device with make config CONFIG_PCIACCESS=n, which using DPDK PCI probe.

When building with make config CONFIG_PCIACCESS=n, e.g. adopting DPDK pci probe, it can't attach nvme device. Does anyone make it work with DPDK rte_eal_pci_probe() on uio_pci_generic?

It seems DPDK try to parse maps folders under /sys/class/uio_pci_generic, and they don't exist. So DPDK skip mmap PCI BARs.

In pci_uio_map_resource(), nb_maps gets 0 from pci_uio_get_mappings(). And it skip any mmap for uio deivce. That results in nvme_attach() failure due to NULL virtual address in mem_resource[], later.

/* collect info about device mappings */
nb_maps = pci_uio_get_mappings(dirname, uio_res->maps,
RTE_DIM(uio_res->maps));
if (nb_maps < 0) {
rte_free(uio_res);
return nb_maps;
}

With default CONFIG_PCIACCESS=y, no such problem with the same setup.
Because libpciaccess probe pci by parsing /sys/bus/0001:001:00.0/resource instead of maps folders under /sys/class/uio_pci_generic

All tests are based on example app nvme_manage.

nvme.c: 458:spdk_nvme_probe: ***ERROR*** NVMe trtype 1 not available

I ran into this problem with NVMeF initiator version 16.12 when running perf example against Linux NVMeF target. I had CONFIG_RDMA?=y set in the CONFIG file. To resolve this, I've added #define SPDK_CONFIG_RDMA to nvme_internal.h and rebuilt.

fio_plugin.c:50:25: fatal error: config-host.h: No such file or directory

I followed the introduction spdk/examples/nvme/fio_plugin/README, but meet the following ERROR:

fio_plugin.c:50:25: fatal error: config-host.h: No such file or directory
 #include "config-host.h"
                         ^
compilation terminated.
make[3]: *** [fio_plugin.o] Error 1
make[2]: *** [fio_plugin] Error 2
make[1]: *** [nvme] Error 2
make: *** [examples] Error 2

Wrong nvme discover command syntax in nvmf getting started doc

nvme discovery -t rdma -s 192.168.100.8 -s 4420
the correct one:
nvme discover -t rdma -a 192.168.100.8 -s 4420

http://www.spdk.io/doc/nvmf_getting_started.html

NVMe bdev name is not persistent

I found NVMe bdev name is not persistent. When I started iscsi_tgt, a NVMe device on 0000:02:00.0 had Nvmp0n1p0. Then I restarted iscsi_tgt, the NVMe device on 0000:02:00.0 changed to Nvme1n1p0. I think bdev name should be persistent.
NVMe spec defines IEEE Extended Unique Identifier (EUI64) in Identify Namespace Data Structure. I think we can use EUI64 as a persistent name for NVMe. What do you think?

Mac OS X Support

Requested by @harshavardhana.

Nvmf: Make session/connection information accessible through public API

I have not found a way to retrieve any information about connections on connect/disconnect callbacks for spdk_nvmf_create_subsystem. It would be nice If one could retrieve the host or if already connected, the session on the connect callback and on disconnect the session the connection belongs to respectively.

Compiling error

make DPDK_DIR=./dpdk-2.1.0/x86_64-native-linuxapp-gcc
== spdk/lib (all)
== spdk/lib/memory (all)
CC vtophys.o
vtophys.c:40:24: fatal error: rte_config.h: No such file or directory
#include "rte_config.h"
^
compilation terminated.
make[2]: *** [vtophys.o] Error 1
make[1]: *** [memory] Error 2
make: *** [lib] Error 2

NvmeF RDMA avoid memory copy

To my understanding, in the current RDMA NVMeF implementation data operations require a copy (cf. nvme_rdma_copy_mem in https://github.com/spdk/spdk/blob/master/lib/nvme/nvme_rdma.c). My suggestion is to enhance the API with either:

1. Function to allocate memory with a specific controller -> memory is registered during allocation
2. Function to (de-)register already allocated memory with a specific controller
3. Register buffers on the fly in read/write functions and cache mrs

to allow buffers to be directly used on data operations.

"Ceph rbd" information should be hidden when use "lsscsi"

When I use spdk iscsi to create a lun for a ceph image, then discovery, login....

When I input "isscsi", it will display "INTEL" and "Ceph rbd":
[14:0:0:0] disk Intel Ceph rbd 0001 /dev/sdc

I looked up the usage of isscsi, it says:

_"iSCSI

The lsscsi command won't show anything iSCSI related on a machine with a iSCSI target running (either as a kernel driver or as a user space daemon). The following output is from a machine running a iSCSI initiator with a session established:
$ lsscsi
[2:0:0:0] disk IET VIRTUAL-DISK 0 /dev/sda"_

I tried the well-known open-source tgtd, which supports librbd also, the result is same as the usage of iscsi.

In my opinion, at last, we shouldn't display "Ceph rbd": the user doesn't need to care which storage software in the back end. Only one thing is: OK, I get a new sdX now. The admin may want to know which image/pool this sdx maps to, and we need to supply another tool(may be enhance rpc.py) to satisfy. "

@dverkamp-intel @optimistyzy,

What is your opinion? If you agree, I could pull a request to resolve it.

Thanks
Pan

SSDs with native SGL format commands support

Any SSD devices in the market today with SGL commands format support additionally to PRP?

Thanks

Cannot add new device(LUN) into an existing iSCSI target.

A new ISCSI target must be created for a new device. This behavior doesn't make sense.

cannot build fio plugin

Hi

I tried to build spdk package with fio plugin.

1. I followed the instruction of spdk/examples/nvme/fio_plugin/README.md and build fio first.
   (1) git clone https://github.com/axboe/fio
   (2) cd fio && git checkout fio-2.8
   (3) ./configure && make
   (4) add following line in ../dpdk-16.07/config/defconfig_x86_64-native-linuxapp-gcc
   EXTRA_CFLAGS=-fPIC
2. I run "make DPDK_DIR=../dpdk-16.07/x86_64-native-linuxapp-gcc/" to build spdk and find the following error message.

LINK examples/nvme/fio_plugin/fio_plugin
/usr/bin/ld: /root/intel_spdk_dpdk/dpdk-16.07/x86_64-native-linuxapp-gcc/lib/librte_eal.a(eal.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/root/intel_spdk_dpdk/dpdk-16.07/x86_64-native-linuxapp-gcc/lib/librte_eal.a(eal.o): could not read symbols: Bad value
collect2: error: ld returned 1 exit status

How can I fix this??

BRs
Edward

libiscsi test suite can cause double free in spdk iSCSI target

Description of the problem
Repeatedly running libiscsi's test suite against a spdk memory based iSCSI target resulted in a double free.

Step to reproduce:

1. Create an iSCSI configuration at app/iscsi_tgt/iscsi.conf that includes the following:

[Malloc]
  # Number of Malloc targets
  NumberOfLuns 1
  # Malloc targets are 128M
  LunSizeInMB 128
  # Block size. Default is 512 bytes.
  BlockSize 512

[InitiatorGroup1]
  InitiatorName ALL
  Netmask 127.0.0.1/16

[TargetNode1]
  TargetName disk1
  TargetAlias "Data Disk1"
  Mapping PortalGroup1 InitiatorGroup1
  UseDigest Auto
  # Use the first malloc target
  LUN0 Malloc0

  QueueDepth 128

2. Run
   sudo app/iscsi_tgt/iscsi_tgt -c app/iscsi_tgt/iscsi.conf
3. After the target is started run
   iscsi-test-cu -V -t SCSI.Read6 iscsi://127.0.0.1/iqn.2016-06.io.spdk:disk1/0
4. When step 3 hangs press Ctrl-C.
5. Run
   iscsi-test-cu -V -t SCSI.Read6 iscsi://127.0.0.1/iqn.2016-06.io.spdk:disk1/0

Expected result:
Test suite not to hang after 3? iSCSI target not to crash after 5?

Actual result:
After 3 libiscsi's test suite hangs on

    Send READ6 (Expecting CHECK_CONDITION) LBA:261985 blocks:160
    [FAILED] READ6 failed with wrong sense. Should have failed with ILLEGAL_REQUEST(0x05)/LBA_OUT_OF_RANGE(0x2100) but failed with Sense: (null)(0x00)/(0x0000)

    Send READ6 (Expecting CHECK_CONDITION) LBA:261984 blocks:161

After 5 the iSCSI target prints this:

iscsi.c:1540:spdk_iscsi_op_login_initialize_port: ***ERROR*** InitiatorName is empty
iscsi.c:4247:spdk_iscsi_execute: ***ERROR*** iscsi_op_login() failed
conn.c:1214:spdk_iscsi_conn_handle_incoming_pdus: ***ERROR*** spdk_iscsi_execute() fatal error on NULL(NULL)
iscsi_subsystem.c: 493:spdk_put_pdu: ***ERROR*** Negative PDU refcount: 0x7f7d825716c0
*** Error in `app/iscsi_tgt/iscsi_tgt': double free or corruption (!prev): 0x00000000024aaa50 ***

How reproducible is the problem?
The problem is reproducible every time.

Version information:
libiscsi commit 3a598ea8465c8204688c6ccbc6d74003342f6651
spdk commit b8da09b
dpdk-16.11

Error building SPDK 16.12 NVMeF initiator

I ran into the following minor error when building SPDK 16.2 NVMeF initiator on the mainline Linux kernel 4.8 with gcc 4.4.7:

nvme_rdma.c:1214: warning: no previous declaration for ‘nvme_rdma_ctrlr_construct’
nvme_rdma.c:1214: error: conflicting types for ‘nvme_rdma_ctrlr_construct’
nvme_rdma.c:1157: note: previous implicit declaration of ‘nvme_rdma_ctrlr_construct’ was here

To resolve this, I've moved nvme_rdma_ctrlr_construct function above nvme_rdma_ctrlr_scan, where it's first called.

identify failed during hotplug running

identify sample program failed when I ran it during hotplug sample program running.

Error message:
nvme_pcie.c: 531:nvme_pcie_ctrlr_allocate_bars: ERROR nvme_pcicfg_map_bar failed with rc 0 or bar (nil)
nvme.c: 313:nvme_ctrlr_probe: ERROR Failed to construct NVMe controller
EAL: Error - exiting with code: 1
Cause: Requested device 0000:05:00.0 cannot be used

How to reproduce:

1. Run hotplug
   # ./example/nvme/hotplug/hotplug -t 600

2. Remove a NVMe device
   # udevadm trigger -c remove -s uio /sys/devices/pci0000:00/0000:00:03.2/0000:05:00.0/uio/uio3

3. Add the removed NVMe device
   # udevadm trigger -c add -s uio /sys/devices/pci0000:00/0000:00:03.2/0000:05:00.0/uio/uio3

4. Just after step 3, run identify
   # ./example/identify/identify

I think this happens when the secondary process pick up an add event before the primary process.

Unrecovered read error -

Hi
spdk's perf application ran fine for a while but since recently I am getting this:

Created task_pool
Initializing NVMe Controllers
associate_workers_with_ns()
Associating INTEL SSDPEDMW400G4 (CVCQ514500A2400AGN ) with lcore 0
Starting thread on core 0
READ sqid:1 cid:113 nsid:1 lba:95459283 len:1
UNRECOVERED READ ERROR (02/81) sqid:1 cid:113 cdw0:0 sqhd:00db p:1 m:1 dnr:0

I tried to reformat the device via nvme-cli which fails as well:

sudo ./nvme format /dev/nvme0n1 --lbaf=03
ioctl: Interrupted system call

any ideas?

Unable to run IOs using the fio plugin

We were in the process of evaluating SPDK on one of our NVMe devices. Was able to build the binaries for SPDK but not able to direct IOs to a particular device.

I am trying to look deeper to figure out the solution, but i thought it would be a good idea to ask the community as well.

Configuration details:
OS: CentOS (4.8.5)
SPDK: 16.12
DPDK: 16.11
Processor: Intel i7-5960X (VT-x supported)

After building and loading SPDK driver, I do not see any block devices. When i issue IO using the device node name 0000:0a:00.0/1 or just 0000:0a:00.0, test errors with invalid file name. How should i initiate IOs to a specific device?

Job file:
[fio_test]
filename=0000:0a:00.0
rw=read
ioengine=./example/nvme/fio_plugin/fio_plugin
iodepth=8
bs=128k
nubjobs=1
offset=1000
size=1M

Am i missing something?

When i initiate IOs, without specifying the filename, the IOs seem to go through. I also see error "could not initialize dpdk" when i increase the number of threads.

I read the documentation about only one thread being supported, I was not able to figure out why so? Any leads on how i should approach to change that?

I was not able to find any documentation related to the above, can someone point me to it if there is any?

Add an fio plugin for the NVMe driver

Create an FIO plugin that allows the tool to run directly against the SPDK NVMe driver.

Can a LUN be added to more than one spdk_scsi_dev ?

Hi,

From the code it looks like we can not add a lun (i.e a bdev like /dev/ram0) to more than one spdk_scsi_dev. But the spdk_scsi_dev_add_lun function in lib/dev.c is not checking for return status of spdk_scsi_lun_claim(lun) before updating dev and id fields in spdk_scsi_lun structure. Because of this dev and id values of previous spdk_scsi_dev are overwritten by new spdk_scsi_dev and when user tries to add lun to second spdk_scsi_dev it succeeds.

Since most of the other code depends on lun->dev field like for example spdk_bdev_scsi_report_luns, wrong values are returned.

Our iscsi.conf

[TargetNode1]
TargetName disk1
Mapping PortalGroup1 InitiatorGroup1
LUN0 Malloc0
LUN1 Malloc1

[TargetNode2]
TargetName disk2
Mapping PortalGroup2 InitiatorGroup2
LUN0 Malloc1 # Please note Malloc1 bdev reused

Can some one please confirm the expected behaviour ?

Thanks & Regards,
Naresh