kaigai/nvme-kmod

A kernel module to support SSD-to-GPU direct DMA

nvme-kmod's Issues

Write protection on the SSD blocks during the READ

Right now, we have no write protection mechanism on SSD blocks during the execution of SSD-to-GPU Direct DMA.
To avoid transferring corrupted blocks to GPU, we need to have a mechanism to prevent write operation to the blocks we try to read asynchronously, until end of DMA.
One potential solution is POSIX flock (mandatory mode). However, it seems to me a bit larger than necessity because it also prevents buffered i/o also. All we want to keep is the contents of the block during the DMA execution.

It is sufficient to prevent write to block on more lower layer.

Support for AMD GPUs via ROCM (ROCnRDMA)

Hi,
just wondering how difficult and if posible to support AMD GPUs via new ROCM ecosystem..
See: https://github.com/RadeonOpenCompute/ROCnRDMA
so it seems to support also RDMA P2P copies even without profesional cards unlike Nvidia so maybe works with consumer line Radeons..

Broken link

Hello,

The pointer to the new repository gives a 404 error.
https://github.com/pg-strom/nvme-strom

Could you please give me some direction for the newer version of this repository?

Thanks

RAID-0/1 Partition Support

Read striping around RAID-0/1 devices potentially improves the throughput to load data blocks to GPU devices than a single raw-device configuration.
I'm now under the investigation for "md" driver for hacking.

Potential Dead-Locking around find_lock_page()

Due to the size mismatch between PAGE_SIZE (=4KB) and CHUNK_SIZE (=8KB; in case of PostgreSQL), we cannot lock all the pages in a particular chunk.
If a concurrent task gets lock of a page then gets one more lock of the prev page in reverse order, NVMe-Strom will wait for the page lock forever.
To avoid it, we must have something like "find_trylock_page" to detect concurrent page access, and retry locks of entire chunks if unavailable to get lock after the reasonable timeout (likely, a few yeild() or schedule() calls)

Is this or something similar possible using OpenCL?

And if it is possible, how can it be done?

How to run NVMe-Strom + PG-Strom.

Mr Cho put the commented below on my blog article.

---

Hi Kaigai,

Thanks for the English post. I've guessed what you're doing so far, and now I can understand it clearly.
Now I'm interested in GPU-SSD DMA for my research, so trying to repeat your experiments these days with a benchmark environments as similar as yours:
HW: i7-6700k CPU, 32GB RAM, Samsung NVMe SSD 172X, and Quadro M4000
SW: CentOS 7 3.10.0-327.28.3.el7.x86_64, CUDA 7.5 + NVIDIA driver 352.39

As I understood as a newbie in kernel programming (I'm a computer vision guy), below are the steps I have to take to repeat your experiments (please correct me if there's something wrong).

1. git nvme-strom then update the kernel
2. git ssd2gpu, compile, then "insmod nvme-strom.ko"
3. run "nvme_test targetfile"

I tried the step2, git clone ssd2gpu -> compille successfully -> "insmod nvme-strom.ko",
but it spits out a error message like below:
cho@bench ssd2gpu]$ sudo insmod nvme_strom.ko
insmod: ERROR: could not insert module nvme_strom.ko: Unknown symbol in module
cho@bench ssd2gpu]$ dmesg | tail -2
[ 6636.226839] nvidia 0000:03:00.0: irq 149 for MSI/MSI-X
[ 7419.227485] nvme_strom: Unknown symbol async_tx_submit (err 0)

Now I'm guessing the step1 has not been made successfully. What I did in step1 is
git nvme-strom -> download kernel source rpm "kernel-3.10.0-327.28.3.el7.src.rpm" -> run "base-upgrade.sh kernel-3.10.0-327.28.3.el7.src.rpm"
This generated nvme-strom.upgrade.diff file, but from here I have no idea what I need to do to proceed to the stpe 2.

Thanks for your time and interesting posts.

Is a Quadro or Tesla GPU necessary?

Would it work with GTX GPUs?
If not, what is the reason why it won't work with GTX GPUs?

ioremap sometimes too slow

1. use nvidia_p2p_get_page get physical pages
2. use ioremap to map nvidia physical page to kernel virtual address
3. use memcpy_toio copy data from kernel to gpu

Sometimes memcpy_toio too slow, it costs 80ms to transfer 600KB data to gpu.

Please help me! How to direct access nvidia physical memory from kernel module?