Streaming upload support `.pipe()`

[veliovgroup]

Streaming upload support `.pipe()`

[dr-dimitru]

Sorry for huge delay, this is implemented in v1.5.1. Please, see piping docs

[dr-dimitru]

Could please explain what do you mean?
Cause video/audio streaming is supported.
And any file upload works like readStream

getStreamSomehow().pipe(streamOperation).pipe(Files)

basically, a way to pipe data to the file collection

[dr-dimitru]

On the client?

Yeah

Basically, a way to expose that stream, maybe a prepareInsert that returns the writable stream

[dr-dimitru]

Cold you please leave few use cases here?

on-the-fly encryption/media encoding is what i'm looking for

[dr-dimitru]

@NROL39 This is only place suitable for your requirements: https://github.com/VeliovGroup/Meteor-Files/blob/master/files.coffee#L707 , so we can add additional callback which will has arrayBuffer or Uint8Array as argument, but it will be just small chunk of whole file, are you sure you will be able to encode file by pieces?

Alternatively I can suggest separate modifier function which will take whole file and then return modified version this ready for upload, but will it be efficient?

Well, loading a 5GB file to memory and operating on it is not feasible on mobile/laptop/pretty-much-any-non-workstation-grade-computer, being able to pipe the data through a modifier function means you only need to hold the current chunk in memory.

[dr-dimitru]

@NROL39 this is what exactly I'm talking about. Please, confirm what it is possible to encode file - chunk by chunk, and encoder doesn't need the whole file.

it is, i've been using it for a while (although what im currently doing is streaming the file through the encoder then rebuilding the file from data chunks from it)

[dr-dimitru]

within this package? right?
If so, could you copy-paste here some code for further testing

using the filestream module for FileReadStream and FIleWriteStream
and a browserify'd version of node.js' crypto library for encryption

key = 'supersecretkey'
iv = 'supersecretiv'
cryptor = crypto.createCipher('aes-128-cbc', key, iv)
fstream = new FileReadStream(file)
fstream.pipe(cryptor).pipe(streamUploadFunction)

where file is a File object
and streamUploadFunction being a function that accepts UInt8Arrays and processes them for download (what we're missing), for now what i'm doing is piping the chunks to a new File object (basically rebuilding the file in memory

fstream.pipe(cryptor).pipe(new FileWriteStream).on('file',callbackFunction)

[dr-dimitru]

As @ghost is removed user, I close this issue