adhocore/tusc.sh

tus 1.0.0 client protocol implementation for bash. Resumable large file upload to Tus sever from terminal using bash script

adhocore/tusc.sh

tusc is tus 1.0.0 client protocol implementation for bash.

tusc lets you upload big files to servers supporting tus protocol right from your terminal.

If anything goes wrong, you can rerun the command to resume upload from where it was left off.

Fun Fact: Git LFS also supports tus.io protocol.

Installation

# Requirements:
# jq
sudo apt install jq -y

curl -sSLo ~/tusc https://raw.githubusercontent.com/adhocore/tusc.sh/main/tusc.sh
# for global binary
chmod +x ~/tusc && sudo ln -s ~/tusc /usr/local/bin/tusc
# OR, for user binary
chmod +x ~/tusc && mv ~/tusc ~/.local/bin/tusc

System Requirements

• base64
• curl
• dd
• grep
• mktemp
• readlink, realpath
• sha1sum, sha256sum, seq, sleep
• tr

Donot worry, in a typical UNIX flavored system these are likely to be there already.

Usage and Examples

  tusc v0.5.0 | (c) Jitendra Adhikari
  tusc is bash implementation of tus-client (https://tus.io).

  Usage:
    tusc <--options> -- [curl args]
    tusc <host> <file> [algo] -- [curl args]

  Options:
    -a --algo      The algorigthm for key &/or checksum.
                   (Eg: sha1, sha256)
    -b --base-path The tus-server base path (Default: '/files/').
    -c --creds     File with credentials; user and pass in shell syntax:
                     USER="my_user"
                     PASS="my_pass"
    -C --no-color  Donot color the output (Useful for parsing output).
    -f --file      The file to upload.
    -h --help      Show help information and usage.
    -H --host      The tus-server host where file is uploaded.
    -L --locate    Locate the uploaded file in tus-server.
    -S --no-spin   Donot show the spinner (Useful for parsing output).
    -u --update    Update tusc to latest version.
       --version   Print the current tusc version.

  Examples:
    tusc --help                           # shows this help
    tusc --update                         # updates itself
    tusc --version                        # prints current version of itself
    tusc    0:1080    ww.mp4              # uploads ww.mp4 to http://0.0.0.0:1080/files/
    tusc -H 0:1080 -f ww.mp4              # same as above
    tusc -H 0:1080 -f ww.mp4 -- -Lv       # same as above plus sends -Lv to curl command
    tusc -H 0:1080 -f ww.mp4 -a sha256    # same as above but uses sha256 algo for key/checksum

If you want to parse the output of tusc, pass in -C (no color) and -S (no spin) flags. Eg:

# Locate the URL of a file and download it
wget $(tusc -H 0:1080 -f ww.mp4 -L -S -C | cut -c 6-999) -O ww.mp4.1

Authentication

If your tusd server requires special header or token for auth, just pass in [curl args]:

tusc -H 0:1080 -f ww.mp4 -b /store/ -- -H "'Authorization: Bearer <token>'" -H "'x-key: value'"

In fact you can pass in anything after -- as extra curl parameter.

Preview

See tusc in action with debug mode where the upload is aborted frequently with Ctrl+C interrupt.

Debugging

To print the debugging information pass in DEBUG=1 env like so:

DEBUG=1 tusc 0:1080 ww.mp4

To print the lines of script as they are executed, create a debug file:

touch ~/.tus.dbg

To revert the above step, just remove the debug file:

rm ~/.tus.dbg

Trying Out

To get hands on in local machine, you can install tusd server.

Then,

# run tusd server (http://0.0.0.0:1080)
tusd -dir ~/.tusd-data > /dev/null 2>&1 &
# start uploading large files
DEBUG=1 tusc --host 0:1080 --file /full/path/to/large/file

# for tusd v2 (http://0.0.0.0:8080)
tusd -upload-dir ~/.tusd-data > /dev/null 2>&1 &
DEBUG=1 tusc --host 0:8080 --file /full/path/to/large/file

While upload is in progress, you can force abort it using Ctrl+C.

Then resume upload again:

DEBUG=1 tusc --host 0:1080 --file /full/path/to/large/file

It should start from where it last stopped.

You can check the uploaded files like so:

ls -al ~/.tusd-data

Contributors

• adhocore - Lead Developer
• tonk - Credential support
• Wouter van Hilst - Chunked upload

tusc.sh's Issues

Progress bar

Missing json

I'm probably doing something wrong, but I tried following the simple startup and I'm getting:

realpath: /home/admin/.tus.json: No such file or directory
./tusc.sh: line 84: $TUSFILE: ambiguous redirect

I'm on an embedded Linux distro, but I installed jq just fine and I'm pretty sure all the other dependencies are in place.

Header may be lowercased

Show location of uploaded file

• when upload finished or aborted
• anytime via tusc --locate --file /the/file

Ability to update tusc

• tusc --update

Tus server replied with empty location

I've tried running on a tus server that works with the js upload library.
However I keep on getting an error with this script
DEBUG=1 ./result/bin/tusc -H https://rustus.sassy.technology -f ../../Downloads/ATDP
_Bitval.pdf -b /files/
HOST : https://myhost
HEADER: /var/folders/m4/1cnl1mnn6qbcnjr7pyshnfd40000gn/T/tus.kJhxpdj8xv
FILE : ATDP_Bitval.pdf
SIZE : 63191
KEY : 8ba20e8bd53cabb96c1ee16fac581125b22f7f12
CHKSUM: sha1 OGJhMjBlOGJkNTNjYWJiOTZjMWVlMTZmYWM1ODExMjViMjJmN2YxMg==

curl -sSLD /var/folders/m4/1cnl1mnn6qbcnjr7pyshnfd40000gn/T/tus.kJhxpdj8xv -H 'Tus-Resumable: 1.0.0' -H 'Upload-Length: 63191' -H 'Upload-Key: 8ba20e8bd53cabb96c1ee16fac581125b22f7f12' -H 'Upload-Checksum: sha1 OGJhMjBlOGJkNTNjYWJiOTZjMWVlMTZmYWM1ODExMjViMjJmN2YxMg==' -H 'Upload-Metadata: filename QVREUF9CaXR2YWwucGRm' -X POST https://myhost/files/
Tus server replied with empty location. Try changing --base-path param.
✖ Unfinished upload, please rerun the command to resume.

I'm not sure where the problem comes from.

How to use other Authentication protocol?

My TUS server implementation use a token base authentication. I have to fill my token in a dedicated HTTP Header.

How can I do that with tusc?

tus version shows as 0.8.1

when doing tusc --help
When you have a moment if you want to update it to 1.0.0 ?
not a big deal though.
Maybe you can make another github release with the latest pr merged as well.
Thanks a lot for making this!

Authentication integration with the tusd server

Firstly, great job. This is awesome!

Can you please provide an example of how to use authentication (username and password) on the tusd server ( I believe this is now done using hooks) and how this would integrate with your code.

I was so happy to see that you added auth into your code.

Can you please provide some guidance as to how I can use your code with the auth and what I need to do to make it use the hooks with the tusd backend.

Thanks in advance.

TUS config cache Location must be host specific

currently ~/.tus.json caches location like

{
  "sha1key": {
    "location": "http://somehost/files/path"
  }
}

but this is problematic as tusc is meant to upload a file to many tusd servers as we dont know which server this location data is for.

the solution is to namespace location by host eg:

{
  "sha1key": {
    "lochost1": "http://host1/files/path1",
    "lochost2": "http://host2/files/path2"
  }
}

curl: (6) Could not resolve host: Bearer | "You can pass in anything after -- as extra curl parameter"

sure... EVERYTHING! sadly even the example doesn't seem to work

and yes, all dependencies are there.