tus-js-client extension support

[tus]

tus-js-client extension support

[janko]

FWIW, here is a function I came up with for generating an MD5 hash of a File object using spark-md5 and chunked-file-reader:

function fileMD5 (file) {
  return new Promise(function (resolve, reject) {
    var spark  = new SparkMD5.ArrayBuffer(),
        reader = new ChunkedFileReader();

    reader.subscribe('chunk', function (e) {
      spark.append(e.chunk);
    });

    reader.subscribe('end', function (e) {
      var rawHash    = spark.end(true);
      var base64Hash = btoa(rawHash);

      resolve(base64Hash);
    });

    reader.readChunks(file);
  })
}

The chunked-file-reader is probably not going to be necessary, since tus-js-client is splitting the File object into chunks of Blob objects which are already loaded whole into memory.

For files of 20MB this takes about 600ms on my machine. I don't expect anyone to use chunks larger than 20MB, so I think it's a pretty small overhead. I haven't compared spark-md5 with other alternatives, but spark-md5 claims to be very fast.

[Acconut]

Well, there are different reasons why these extensions haven't been implemented yet:

• The Checksum extension requires that the checksum is calculated inside the browser. While this is totally doable today, it's particularly expensive and time intensive for bigger files and on mobile devices. Of course, we could utilise the new Web Crypto API, which probably offers better performance and security, but you could argue whether it has reached critical mass yet (see http://caniuse.com/#feat=cryptography).
• The Concatenation extension is mostly meant for parallel uploads where you need to utilise multiple HTTP connections. In most cases, this does not apply to the environment of the browser but it can also be used for different things. Supporting this is basically just adding some headers.
• The Termination extension is pretty much the same as with the concatenation. Just adding a new HTTP method and there you go. :)

To summaries, there is no real reason why extension support is not high right now, except the (maybe) missing needs and technical details. A user could build the required code on top of tus-js-client but is not supposed to do so. I hope to improve the situation soon.

[matthoskins1980]

@Acconut makes total sense - and please don't take my question as criticism - this protocol definition is exactly what we (the webdev community) needed!

I want to make sure I understand the Concatenation extension - let me explain my use case and see if it fits. I want to give my users the ability to upload medium-to-large video files (in excess of 350MB) over a mobile hotspot. The browser will be on a laptop (not mobile). In order to maximize network performance - I like to make the upload multi-threaded. Essentially - I'd like to upload one file over 3 to 5 HTTP connections. Could concatenation be used for this?

[Acconut]

Yes, that's one of the use cases we designed the Concatenation extension for.

[kvz]

@Acconut I like your explanations in #29 (comment), would it make sense to cement (some of) those in README.md?

[Acconut]

@janko-m Those are some promising numbers, thanks for sharing!

[janko]

Another update, the default chunk size for ChunkedFileReader is 256 KB, which I think is bit too small. I tried bumping the chunk size to 2MB, which I think is very reasonable, and for me that sped up hashing from 600ms to 200ms (for a 20MB file).

So, I think we should be good regarding the performance of calculating a checksum, though it would probably be good to also test the speed on mobile devices.