Filtering, Max Files and Supporting URLs

[doom-fish]

Filtering, Max Files and Supporting URLs

[jaruba]

I also found the readme of unzip-stream interesting: https://www.npmjs.com/package/unzip-stream#parse-zip-file-contents

As it also has a method of giving the user of the module control over the traversal of the archive's file list.

[jaruba]

@1313 sorry for the late reply, i have been hunting for a new apartment lately

When I think of it I think we can optimize it further and calculate the chunks with some arithmetics.

That would be great, I have many test cases if you need anything tested, and we are also willing to assist with development for improving the module. Should we wait to see what results your current attempts lead to? Or do you have any details of how you think some features should be best implemented?

[1313]

Hello 👋 so nice to hear that you find use of this library!

Hehe I had the same discovery that most media related cases uses rar just for file splitting. As a retired warez scener I know the reasons for splitting files like this but it makes less sense from the outside. Anyhow, nice to be able to stream things directly without having to have all the content locally.

I am a bit busy at the moment, but your suggestions are reasonable and sound. I will assess things more closely tomorrow.

Happy to receive PRs.

Cheers and thanks for the kind words!

[1313]

When I think of it I think we can optimize it further and calculate the chunks with some arithmetics. As we know the file size and the intermediate chunks should have a constant size for file heads.

I'll do some digging. Filtering is not a bad idea either, especially with a lot of smaller files.

[1313]

Haha when revisiting the code I see that the optimisation is already there.

Sorry for a late reply here as well. Short on free time with two toddlers to run after. I hope the apartment hunt bares fruit.

Anyhow, I think a filtering solution would be great, if you want to give it a shot I'm happy to review a PR and we can take the discussion from there.

Cheers

[jaruba]

@1313 tell me what you think: #30

[jaruba]

although off-topic, another interesting case i ran into is the way that the FileMedia interface works: https://github.com/doom-fish/rar-stream?tab=readme-ov-file#innerfile-api

it expects createReadStream() to be synchronous, which only creates an issue if I try to use node-fetch instead of needle, as the expected way of using node-fetch would be:

const createReadStream = async () => {
  const fetch = require('node-fetch');
  const resp = await fetch(url, opts);
  return resp.body; // this is a stream
}

so node-fetch can only be used in an async function, using await file.createReadStream() should work for both sync and async cases.

this is outside of our need, i just thought that using node-fetch would have been interesting because it can easily be swapped with the browser's own HTML5 fetch, and there is a good chance that rar-stream could be browserified to unpack and play a video straight in the browser when combined with something like: https://www.npmjs.com/package/videostream

just something to think about 😄

there is now a PR for this change here: #29

[1313]

I'll look into making the lib browser ready as well. I suspect that the binary dependency used for reading the headers requires node.