saldl/saldl

A lightweight well-featured CLI downloader optimized for speed and early preview.

A multi-platform command-line downloader optimized for speed and early preview, based on libcurl.

saldl splits a download into fixed-sized chunks and download them in-order with multiple concurrent connections. Classic modes (single chunk single connection, no. of chunks = no. of connections) are also available as options.

More than 57 command-line options are available to customize saldl's behavior.

Documentation

Detailed documentation is available in the form of a manual page.

Contact

Users Chat & Questions

FlashGot integration

Check out the README and the scripts in the flashgot directory.

Screenshot

   -s4m => set chunk_size to 4 MiB.
   -c4  => use 4 concurrent connections.
   -l4  => download the last 4 chunks first.

Dependencies

• Runtime Dependencies

• libcurl >= 7.55

• libevent + libevent_pthreads >= 2.0.20

• Build Dependencies

• GCC or Clang

• python (for waf)

• Optional Build Dependencies

• git (to get the current version).

• asciidoc (to build the manual).

Note: docbook-xsl should be installed too if it's not already a dependency of asciidoc. While this is theoritically not necessary, falling back to getting XSL stylesheets directly from SourceForge seems to be unreliable nowadays.

Build

saldl has been tested on GNU/Linux, NetBSD, FreeBSD, OpenBSD, Mac OSX, and Windows.

saldl should build and run successfully on all POSIX platforms. If anyone runs into any build failures, reporting them would be highly appreciated.

POSIX

• General

Run ./waf --help first and check out the available options.

A typical example would be:

./waf configure --prefix=/usr

./waf

./waf install --destdir=${pkgdir}

• Arch Linux

saldl is available in the AUR.

• Mac OSX

saldl can be installed with Homebrew.

For the latest release, run:

  brew install saldl

For the latest GIT head, run:

  brew install --HEAD saldl

Note: saldl will be linked against system libcurl if the Mac OSX version installed is Yosemite (10.10) or newer. Pass --with-curl if you want to link against the latest libcurl release provided by Homebrew.

Windows

• MSYS2/Cygwin

A PKGBUILD for MSYS2.

Note: Building against the MSYS2 runtime (or Cygwin's) is necessary for saldl to work correctly in mintty.

• MinGW-w64/Native

Experimental binaries are now available in the releases page.

saldl requires a terminal emulator with support for ANSI/VT100 escape sequences (e.g. ConEmu or ansicon).

saldl's Issues

Remove unnecessary null pointer checks

Extra null pointer checks are not needed in the function "sal_dl".

Building on OS X 10.11 with Xcode 8 fails

saldl -v

saldl v34 (https://saldl.github.io)

Copyright (C) 2014-2016 Mohammad AlSaleh.
Free use of this software is granted under the terms of
the GNU Affero General Public License (AGPL).

Built against: libcurl 7.43.0
Loaded: libcurl 7.43.0 (SecureTransport)
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp.

Describe the issue / How to reproduce

Building on OS X 10.11 El Capitan with Xcode 8 fails due to a clock_gettime error. This is due to the fact that the unitary SDK model provides symbols that are defined in the Xcode 8 SDK headers used for both 10.11 and 10.12, but only implemented in the system libraries found on 10.12.

How to reproduce: attempt to compile on OS X 10.11 using Xcode 8.

log output (if needed)

http://pastebin.com/GUPc7QaQ

---

If you have need of any more information I am happy to provide it.

Support for Multiple URLs to download (in order provided)

saldl -v

saldl v35.0.g36e89b3 (https://saldl.github.io)
WARNING: Windows builds of saldl are experimental.

Copyright (C) 2014-2016 Mohammad AlSaleh.
Free use of this software is granted under the terms of
the GNU Affero General Public License (AGPL).

Built against: libcurl 7.52.1
Loaded: libcurl 7.52.1 (OpenSSL/1.0.2j)
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp.

Describe the issue / How to reproduce

For example, if you have a list of URIs, each URI being a download of one 7 second segment of one big file, there's no convenient way with saldl to download that. Aria2c allows you to download multiple URLs by presenting it with a large \n\r separated string as URL with options for each url indented with a TAB character.

Is there any plans for something like this? I don't particularly need the ability to use different settings on different urls, just the normal global settings on all urls.

Perhaps an interface like saldl [OPTIONS] [URL(s)...] e.g. saldl --merge-in-order -c8 --auto-size 1 -D tears-of-steel -o segs_1_and_2.ts http://demo.unified-streaming.com/video/tears-of-steel/tears-of-steel.ism/tears-of-steel-audio_eng=64008-video_eng=401000-1.ts http://demo.unified-streaming.com/video/tears-of-steel/tears-of-steel.ism/tears-of-steel-audio_eng=64008-video_eng=401000-2.ts would download both segments in order, with multiple connections for each segment (if possible).

log output (if needed)

libcurl error 61: Error while processing content unencoding: invalid block type

I tried downloading this quite big archived file:
https://www.cs.cmu.edu/~./enron/enron_mail_20150507.tgz

saldl exits with this:

[fatal] libcurl returned fatal error (61: Error while processing content unencoding: invalid block type) while downloading chunk 3.

In curl and it works fine out of the box.

Eventually I got it working by passing the --no-compress flag. I'm just trying to understand what was going on there...

Compilation fails

Running the ./waf command fails on Ubuntu 14.04 64 bit. ./waf configure --prefix=/usr runs fine btw. Why is this?

chinx@slimshady:~/Downloads/saldl-master$ ./waf
Waf: Entering directory `/home/chinx/Downloads/saldl-master/build'
[ 1/17] Compiling src/progress.c
[ 2/17] Compiling src/events.c
[ 3/17] Compiling src/write_modes.c
[ 4/17] Compiling src/queue.c
[ 5/17] Compiling src/merge.c
[ 6/17] Compiling src/status.c
[ 7/17] Compiling src/resume.c
[ 8/17] Compiling src/ctrl.c
[ 9/17] Compiling src/common.c
[10/17] Compiling src/exit.c
[11/17] Compiling src/log.c
[12/17] Compiling src/utime.c
[13/17] Compiling src/transfer.c
[14/17] Compiling src/saldl.c
[15/17] Compiling src/main.c
[16/17] Compiling saldl.1.txt
[17/17] Linking build/saldl
a2x: WARNING: --destination-dir option is only applicable to HTML based outputs
a2x: ERROR: "xsltproc"  --stringparam callout.graphics 0 --stringparam navig.graphics 0 --stringparam admon.textlabel 1 --stringparam admon.graphics 0  "/usr/local/etc/asciidoc/docbook-xsl/manpage.xsl" "/home/chinx/Downloads/saldl-master/build/saldl.1.xml" returned non-zero exit status 127

Waf: Leaving directory `/home/chinx/Downloads/saldl-master/build'
Build failed
 -> task in '' failed (exit status 1): 
    {task 140543315836368: a2x-man-1 saldl.1.txt -> saldl.1}
['/usr/local/bin/a2x', '-f', 'manpage', '-D.', '../saldl.1.txt']

Build error on musl

saldl -v

The build error happened with both the latest tagged release and the scm version. The log printed is from the master branch.

Describe error

Build error on a musl based system.

Log output

In function 'colorset.constprop',
    inlined from 'status_update_cb' at ../src/status.c:140:7:
../src/status.c:47:7: error: 'strncpy' output truncated before terminating nul copying as many bytes from a string as its length [-Werror=stringop-truncation]
       strncpy(ptr,end,strlen(end));
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/status.c:42:7: error: 'strncpy' output truncated before terminating nul copying as many bytes from a string as its length [-Werror=stringop-truncation]
       strncpy(ptr,invert,strlen(invert));
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/status.c:66:9: error: 'strncpy' output truncated before terminating nul copying as many bytes from a string as its length [-Werror=stringop-truncation]
         strncpy(ptr,ok_color,strlen(ok_color));
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/status.c:62:9: error: 'strncpy' output truncated before terminating nul copying as many bytes from a string as its length [-Werror=stringop-truncation]
         strncpy(ptr,info_color,strlen(info_color));
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/status.c:58:9: error: 'strncpy' output truncated before terminating nul copying as many bytes from a string as its length [-Werror=stringop-truncation]
         strncpy(ptr,warn_color,strlen(warn_color));
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/status.c:54:9: error: 'strncpy' output truncated before terminating nul copying as many bytes from a string as its length [-Werror=stringop-truncation]
         strncpy(ptr,error_color,strlen(error_color));
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/status.c:75:5: error: 'strncpy' output truncated before terminating nul copying as many bytes from a string as its length [-Werror=stringop-truncation]
     strncpy(ptr,end,strlen(end));
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
In function 'colorset',
    inlined from 'status_display' at ../src/status.c:189:3:
../src/status.c:54:9: error: 'strncpy' output truncated before terminating nul copying as many bytes from a string as its length [-Werror=stringop-truncation]
         strncpy(ptr,error_color,strlen(error_color));
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/status.c:75:5: error: 'strncpy' output truncated before terminating nul copying as many bytes from a string as its length [-Werror=stringop-truncation]
     strncpy(ptr,end,strlen(end));

Build failed
 -> task in 'saldl-objs' failed with exit status 1: 
	{task 140300765654488: c status.c -> status.c.1.o}
['x86_64-pc-linux-musl-cc', '-march=native', '-O3', '-pipe', '-Wall', '-Wextra', '-Wmissing-format-attribute', '-Wno-implicit-fallthrough', '-Werror', '-std=gnu99', '-pedantic', '-fPIE', '-I/usr/x86_64-pc-linux-musl/include', '-DSALDL_VERSION="v40.0.gb3c0370"', '-D_FILE_OFFSET_BITS=64', '-D_GNU_SOURCE', '-DSALDL_DEF_NUM_CONNECTIONS=6', '-DSALDL_DEF_CHUNK_SIZE=1048576', '-DHAVE_MKDIR=1', '-DHAVE_CLOCK_MONOTONIC_RAW=1', '-DHAVE_GETTIMEOFDAY=1', '-DHAVE_FILENO=1', '-DHAVE_ISATTY=1', '-DHAVE_IOCTL=1', '-DHAVE_STRFTIME=1', '-DHAVE_ASPRINTF=1', '-DHAVE_STRCASESTR=1', '-DHAVE_STRSIGNAL=1', '-DHAVE_SIGACTION=1', '-DHAVE_SIGADDSET=1', '-DHAVE_MMAP=1', '-DHAVE_LIBEVENT_PTHREADS=1', '-DHAVE_LIBCURL=1', '../src/status.c', '-c', '-o/tmp/paludis/build/net-misc-saldl-scm/work/saldl-scm/build/src/status.c.1.o', '-march=native', '-O3', '-pipe']

     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

can't deal with "&" in url?

saldl -v

Describe the issue / How to reproduce

if there's a "&" in URL, it can't read the whole URL. For example,

it throw out this error.
'fid' is not an internal or external command, nor is it a runnable program or batch file.

log output (if needed)

Request support Metalink

Request support Metalink.

Static linking?

saldl -v

git @ 0926ac1

Describe the issue / How to reproduce

As far as I can tell, it doesn't seem possible to compile saldl statically... It would be really nice to have statically linked binaries available for download because the curl dependency can be hard to manage on some systems (particularly older Debians or even modern Ubuntu). Saldl is particularly useful to me as a time saver but I'm not sure there's much of a net gain if I'm spending all my time just trying to get it to compile...

log output (if needed)

My attempt seemed to almost work until I got to linking... Eek!

Support for input file

I looked the documentation, but didn't find option to input list of urls. Am I missing something?

reserved identifier violation

I would like to point out that identifiers like "_SALDL_COMMON_H" and "_SALDL_STRUCTS_H" do not fit to the expected naming convention of the C language standard.
Would you like to adjust your selection for unique names?

Completion of error handling

Would you like to add more error handling for return values from functions like the following?

• pthread_create ⇒ sal_dl
• strdup ⇒ valid_filename

Users Chat & Questions

This is a meta bug for users questions.

If you have a question. Ask it here.

Homebrew build fails on OSX 10.9.5

$ brew install https://github.com/saldl/saldl/raw/master/OSX/saldl.rb
######################################################################## 100.0%
==> Downloading https://github.com/saldl/saldl/archive/v31.tar.gz
==> Downloading from https://codeload.github.com/saldl/saldl/tar.gz/v31
######################################################################## 100.0%
==> ./waf configure --libcurl-cflags=-I/usr/local/opt/curl/include --libcurl-libs=-L/usr/local/opt/curl/lib -lcurl --prefix=/usr/local/Cellar/saldl/31 --saldl-version=v31
==> ./waf build
Last 15 lines from /Users/user/Library/Logs/Homebrew/saldl/02.waf:
[12/17] Compiling src/utime.c
[13/17] Compiling src/transfer.c
[14/17] Compiling src/saldl.c
[15/17] Compiling src/main.c
[16/17] Compiling saldl.1.txt
  File "/usr/local/opt/asciidoc/bin/a2x", line 76
    print '%s: %s' % (PROG,msg)
                 ^
SyntaxError: invalid syntax

Waf: Leaving directory `/private/tmp/saldl20151108-40246-1xpp3fx/saldl-31/build'
Build failed
 -> task in '' failed (exit status 1):
    {task 4325720920: a2x-man-1 saldl.1.txt -> saldl.1}
['/usr/local/opt/asciidoc/bin/a2x', '-f', 'manpage', '-D.', '../saldl.1.txt']