danishprakash/py-splice

A Python interface to splice(2) system call

py-splice's Issues

Please make buffer size tunable

You use 1 page size buffer for each splice call. Its fine if second splice (to pipe) allows kernel to just mark memory pages as pagecache (thus, its fast). But fails if underiying pip is slow. For example if it belongs to file opened with O_DIRECT.

Simple test to splice data between 1GB files opened with O_DIRECT each:

(2 x SATA3 RAID0 ssd)

4k    buf_size (your default):    60.67s   17MiB/s
64k   buf size:                   13.17s   78MiB/s
256k  buf size:                    9.05s  113MiB/s
1024k buf size:                    5.46s  188MiB/s
4096k buf size:                    3.80s  270MiB/s
8192k buf size:                    3.16s  325MiB/s

What did I do above:

1. Tuned buf_size accrordingly for each run
2. Tuned pipe buf size for both ends (fcntl F_SETPIPE_SZ) for same value as buf_size

Pipe buf is 64k by default in modern linux. Everything above 1M requires CAP_SYS_RESOURCE non-root users. 8M is maximum I was able to test, everything above returns ENOMEM for root.

Thanks!

nbytes and offset should be size_t

subj:

py-splice/splicemodule.c

Line 132 in 50bb8dd

int status = -1, offset = 0, flags = 0, nbytes = 0;