Ask Questions

[facebook]

Ask Questions

[wakeupnuts]

Which will be done faster by wdt : one large file or many small files (equal size in total, without network throttling) and why? Thanks!

[ldemailly]

@wakeupnuts we cut large files into smaller blocks to be sent in parallel so in that sense even a single large file benefits from wdt's multiple flow pipeline

If you have a lot of tiny files (smaller than the blocksize), there is overhead in sending their pathname and opening/reading/writing them, so that's likely to be a bit slower even though it's probably faster than any other way (like using "tar") because we have a really compact and efficient encoding of each header

let me know if that answers your question or if you have any additional ones

[wakeupnuts]

@ldemailly Thank you very much for the answer, it's exactly what I need. 😄

I've been searching for a fast data transfer tool for a while, in order to minimize the time of a daily data-transfer task (between a data host and a few receivers, < 10 files, 50G~70G in total) in a production system. Wdt looks very promising. I'd like to give a try, and come back with more questions.

One more thing, is there a way to limit/configure the transfer speed of wdt? Consider wdt's performance, there might be a chance that wdt-task consume total bandwidth and starves other tasks.

[ldemailly]

Looking forward to more questions and helping you use WDT

Yes WDT includes a Throttler so you don't negatively impact the source or destination machine (or network in between) when they are used for other tasks while transfering. Check the avg_mbytes_per_sec option/flag for instance.

ps: I am traveling / in vacation so will try to not work/reply too much until I am back Aug 10 - but the rest of our team may :)

[ldemailly]

You can now also ask questions on https://gitter.im/facebook/wdt