netromdk/efdl

efdl is an efficient downloading CLI app over HTTP(S)

efdl

efdl, pronounced "Eff-dell", is an efficient downloading command-line application that currently supports transfers using HTTP(S).

It is also possible to use the functionality of efdl in your own program. Just link the libefdlcore static library and use the headers in the include directory.

Requirements

A C++11 compliant compiler (GCC 4.8+, Clang 3.3+ etc.), CMake 2.8.12+, and Qt 5.2+.

Compilation and installation

To compile the source code and link the binaries do the following:

1. Extract source and go into the diretory.
2. mkdir build
3. cd build
4. cmake .. (If you want the shared library then use -DLIBRARY_TYPE=SHARED)
5. make

This produces the efdl binary in the bin directory.

Notice the install prefix when running cmake before, which defaults to /usr/local. If you run sudo make install it will install into the bin directory there. Uninstalling is accomplished with sudo make uninstall.

Program usage

Usage: efdl [options] URLs..
Efficient downloading application.

If URLs are given through STDIN then the positional argument(s) are optional.
Also note that piping URLs will make confirmations default to 'no', if any.

HTTP basic authorization is supported either via --http-user and --http-pass
or by using URLs on the form 'scheme://username:[email protected]/path/'.

Options:
  -h, --help               Displays this help.
  -v, --version            Displays version information.
  -o, --output <dir>       Where to save file. (defaults to current directory)
  -c, --conns <num>        Number of simultaneous connections to use. (defaults
                           to 1)
  -r, --resume             Resume download if file is present locally and the
                           server supports it.
  --confirm                Will ask to confirm to download on redirections or
                           whether to truncate a completed file when resuming.
  --verbose                Verbose mode.
  --dry-run                Do not download anything just resolve URLs and stop.
  --chunks <num>           Number of chunks to split the download up into.
                           Cannot be used with --chunk-size.
  --chunk-size <bytes>     Size of each chunk which dictates how many to use.
                           Cannot be used with --chunks.
  --http-user <user>       Username for HTTP basic authorization.
  --http-pass <pass>       Password for HTTP basic authorization.
  --show-conn-progress     Shows progress information for each connection.
  --show-http-headers      Shows all HTTP headers. Implies --verbose.
  --verify <fmt=hash, ..>  Verify the integrity of the downloaded file(s) using
                           the given hash function and value. Hash functions
                           supported: md4, md5, sha1, sha2-224, sha2-256,
                           sha2-384, sha2-512, sha3-224, sha3-256, sha3-384,
                           sha3-512
  --gen-checksum <fmt>     Generate a checksum of the downloaded file using the
                           given hash function. See --verify for supported hash
                           functions.

Arguments:
  URLs                  URLs to download.

efdl's Issues

No Content-Length then fallback to single conn GET

In some cases the response does not include a Content-Length, so fallback to a single connection HTTP GET. Ranges might or might not be supported so it's best to create the "empty" range [0,0] in this case, which lets the DownloadTask know that it should not include that header.

Failing with docker pkg download

When trying to install the Docker package from GitHub; efdl will fail on different chunks during the download and crash.

% ./bin/efdl https://github.com/boot2docker/osx-installer/releases/download/v1.5.0/Boot2Docker-1.5.0.pkg
Downloading https://github.com/boot2docker/osx-installer/releases/download/v1.5.0/Boot2Docker-1.5.0.pkg
Resolved to https://s3.amazonaws.com/github-cloud/releases/17077232/78eb97be-b13f-11e4-9df5-207a655745c1.pkg?response-content-disposition=attachment%3B%20filename%3DBoot2Docker-1.5.0.pkg&response-content-type=application/octet-stream&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1425268318&Signature=tQ%2BO3rHWj0mUs8bxAwd3mIGi9bI%3D
File size 129.8 MB [application/octet-stream]
Saving to /Users/brandon/git/efdl/build/78eb97be-b13f-11e4-9df5-207a655745c1.pkg
[ 15.4% | 20.0 MB / 129.8 MB @ 341.3 KB/s | chunk 20 / 130 | 5m 29s left ]
Chunk 21 failed on range [20971520, 22020095]
HTTP code: 403
Error: Operation not permitted on remote server.
Aborting..
QThread: Destroyed while thread is still running
[1]    887 segmentation fault  ./bin/efdl

Read file name from Content-Disposition

If available then save file as the "filename" in HTTP header "Content-Disposition" if it is available.

Example:
"Content-Disposition: attachment; filename=Boot2Docker-1.5.0.pkg"

Revert to HTTP GET if chunking is unsupported

In the case that a server does not support chunking then do a single connection HTTP GET instead.

Implement interruption in DownloadTask

It should run a timer that checks whether a request for interruption has been given. If that is the case then quit the event loop and abort the request. And emit interrupted() signal.

Run static analysis

Run clang static analysis checker.

Weird Linux segfault

When a DownloadTask is run then it segfaults randomly inside on Qt. Sometimes it dies in QNetworkRequest req{url};, sometimes in if (!httpUser.isEmpty() && !httpPass.isEmpty()) { or auto *rep = netmgr.get(req);.

And it only happens on Linux. Tested using Qt 5.2 and 5.4.

(gdb) r
Starting program: /mnt/hgfs/netrom/git/efdl/build_linux14/bin/efdl -c 1 --chunk-size 2000000 --show-http-headers --verbose http://download.keyshot.com/keyshot_mac64_5.1.66.pkg
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff3e00700 (LWP 48119)]
Downloading http://download.keyshot.com/keyshot_mac64_5.1.66.pkg
HEAD http://download.keyshot.com/keyshot_mac64_5.1.66.pkg
[New Thread 0x7ffff2fc7700 (LWP 48120)]
[New Thread 0x7ffff27c6700 (LWP 48121)]
CODE 206
HEADERS [x-amz-id-2: OAh5l8fQEy4drSavXJK1R7OLFike8gR5O8tZvZnyw4Gp2M6DBLStna0hC8RNJLkN] [x-amz-request-id: 6124F59586E126FE] [Date: Sun, 22 Mar 2015 19:01:55 GMT] [Last-Modified: Thu, 18 Dec 2014 08:51:06 GMT] [ETag: "6257aa911d5ba667215200c3d2ce1138-8"] [Accept-Ranges: bytes] [Content-Range: bytes 0-0/473279315] [Content-Type: application/octet-stream] [Content-Length: 1] [Server: AmazonS3]
File size 451.4 MB [application/octet-stream]
RESUMABLE
Saving to /mnt/hgfs/netrom/git/efdl/build_linux14/keyshot_mac64_5.1.66.pkg
CHUNK SIZE 1.9 MB
CHUNKS 237
[ Starting up download.. ]
[New Thread 0x7ffff1795700 (LWP 48129)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff1795700 (LWP 48129)]
0x00007ffff7ad5668 in QUrl::operator=(QUrl const&) () from /usr/local/Qt-5.4.0/lib/libQt5Core.so.5

(gdb) bt

0 0x00007ffff7ad5668 in QUrl::operator=(QUrl const&) () from /usr/local/Qt-5.4.0/lib/libQt5Core.so.5

1 0x00007ffff7f175b0 in QNetworkRequest::QNetworkRequest(QUrl const&) () from /usr/local/Qt-5.4.0/lib/libQt5Network.so.5

2 0x0000000000434c18 in efdl::DownloadTask::run (this=0x687110) at /mnt/hgfs/netrom/git/efdl/src/core/DownloadTask.cpp:22

3 0x00007ffff7954f8f in ?? () from /usr/local/Qt-5.4.0/lib/libQt5Core.so.5

4 0x00007ffff6dbe182 in start_thread (arg=0x7ffff1795700) at pthread_create.c:312

5 0x00007ffff70cf00d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

(gdb) f 2

2 0x0000000000434c18 in efdl::DownloadTask::run (this=0x687110) at /mnt/hgfs/netrom/git/efdl/src/core/DownloadTask.cpp:22

22 QNetworkRequest req{url};
(gdb) l
17 void DownloadTask::onProgress(qint64 received, qint64 total) {
18 emit progress(num, received, total);
19 }
20
21 void DownloadTask::run() {
22 QNetworkRequest req{url};
23 req.setRawHeader("Accept-Encoding", "identity");
24
25 // Only set range information if appropriate.
26 qint64 start = range.first, end = range.second;

(gdb) info threads
Id Target Id Frame
(*) 5 Thread 0x7ffff1795700 (LWP 48129) "efdl::DownloadT" 0x0000000000434c18 in efdl::DownloadTask::run (this=0x687110) at /mnt/hgfs/netrom/git/efdl/src/core/DownloadTask.cpp:22
4 Thread 0x7ffff27c6700 (LWP 48121) "Thread (pooled)" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
3 Thread 0x7ffff2fc7700 (LWP 48120) "Qt HTTP thread" 0x00007ffff70c6a09 in __pselect (nfds=10, readfds=0x7fffe4000a78, writefds=0x7fffe4000d08, exceptfds=0x7fffe4000f98,
timeout=, sigmask=) at ../sysdeps/unix/sysv/linux/pselect.c:77
2 Thread 0x7ffff3e00700 (LWP 48119) "Qt bearer threa" 0x00007ffff70c6a09 in __pselect (nfds=7, readfds=0x7fffec000a78, writefds=0x7fffec000d08, exceptfds=0x7fffec000f98,
timeout=, sigmask=) at ../sysdeps/unix/sysv/linux/pselect.c:77
1 Thread 0x7ffff7e86780 (LWP 48115) "efdl" 0x00007ffff79594b0 in QByteArray::append(char) () from /usr/local/Qt-5.4.0/lib/libQt5Core.so.5