kaigai / nvme-kmod Goto Github PK
View Code? Open in Web Editor NEWA kernel module to support SSD-to-GPU direct DMA
License: GNU General Public License v2.0
A kernel module to support SSD-to-GPU direct DMA
License: GNU General Public License v2.0
Right now, we have no write protection mechanism on SSD blocks during the execution of SSD-to-GPU Direct DMA.
To avoid transferring corrupted blocks to GPU, we need to have a mechanism to prevent write operation to the blocks we try to read asynchronously, until end of DMA.
One potential solution is POSIX flock (mandatory mode). However, it seems to me a bit larger than necessity because it also prevents buffered i/o also. All we want to keep is the contents of the block during the DMA execution.
It is sufficient to prevent write to block on more lower layer.
Hi,
just wondering how difficult and if posible to support AMD GPUs via new ROCM ecosystem..
See: https://github.com/RadeonOpenCompute/ROCnRDMA
so it seems to support also RDMA P2P copies even without profesional cards unlike Nvidia so maybe works with consumer line Radeons..
Hello,
The pointer to the new repository gives a 404 error.
https://github.com/pg-strom/nvme-strom
Could you please give me some direction for the newer version of this repository?
Thanks
Read striping around RAID-0/1 devices potentially improves the throughput to load data blocks to GPU devices than a single raw-device configuration.
I'm now under the investigation for "md" driver for hacking.
Due to the size mismatch between PAGE_SIZE (=4KB) and CHUNK_SIZE (=8KB; in case of PostgreSQL), we cannot lock all the pages in a particular chunk.
If a concurrent task gets lock of a page then gets one more lock of the prev page in reverse order, NVMe-Strom will wait for the page lock forever.
To avoid it, we must have something like "find_trylock_page" to detect concurrent page access, and retry locks of entire chunks if unavailable to get lock after the reasonable timeout (likely, a few yeild() or schedule() calls)
And if it is possible, how can it be done?
Mr Cho put the commented below on my blog article.
Hi Kaigai,
Thanks for the English post. I've guessed what you're doing so far, and now I can understand it clearly.
Now I'm interested in GPU-SSD DMA for my research, so trying to repeat your experiments these days with a benchmark environments as similar as yours:
HW: i7-6700k CPU, 32GB RAM, Samsung NVMe SSD 172X, and Quadro M4000
SW: CentOS 7 3.10.0-327.28.3.el7.x86_64, CUDA 7.5 + NVIDIA driver 352.39
As I understood as a newbie in kernel programming (I'm a computer vision guy), below are the steps I have to take to repeat your experiments (please correct me if there's something wrong).
I tried the step2, git clone ssd2gpu -> compille successfully -> "insmod nvme-strom.ko",
but it spits out a error message like below:
cho@bench ssd2gpu]$ sudo insmod nvme_strom.ko
insmod: ERROR: could not insert module nvme_strom.ko: Unknown symbol in module
cho@bench ssd2gpu]$ dmesg | tail -2
[ 6636.226839] nvidia 0000:03:00.0: irq 149 for MSI/MSI-X
[ 7419.227485] nvme_strom: Unknown symbol async_tx_submit (err 0)
Now I'm guessing the step1 has not been made successfully. What I did in step1 is
git nvme-strom -> download kernel source rpm "kernel-3.10.0-327.28.3.el7.src.rpm" -> run "base-upgrade.sh kernel-3.10.0-327.28.3.el7.src.rpm"
This generated nvme-strom.upgrade.diff file, but from here I have no idea what I need to do to proceed to the stpe 2.
Thanks for your time and interesting posts.
Would it work with GTX GPUs?
If not, what is the reason why it won't work with GTX GPUs?
Sometimes memcpy_toio too slow, it costs 80ms to transfer 600KB data to gpu.
Please help me! How to direct access nvidia physical memory from kernel module?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.