doom-fish/rar-stream

Library for "unpacking" and reading files inside rar archives as node Readable streams.

rar-stream's Issues

Filtering, Max Files and Supporting URLs

Hi! We've been playing with your module for the last few days and were hoping to include it in Stremio's local streaming server.

I wanted to share our experience with it. 😄

Specifically this code gave us a bit of trouble:

rar-stream/src/rar-files-package.ts

Lines 55 to 75 in caf7953

	while (fileOffset < rarFile.length - TerminatorHeaderParser.HEADER_SIZE) {
	const fileHead = await parseHeader(FileHeaderParser, rarFile, fileOffset);
	if (fileHead.type !== 116) {
	break;
	}
	if (fileHead.method !== 0x30) {
	throw new Error("Decompression is not implemented");
	}
	fileOffset += fileHead.headSize;
	fileChunks.push({
	name: fileHead.name,
	fileHead,
	chunk: new RarFileChunk(
	rarFile,
	fileOffset,
	fileOffset + fileHead.size - 1
	),
	});
	fileOffset += fileHead.size;
	}

This is used to traverse the archive in order to read the file headers and prepare the file chunks.

We stress tested it with some cases, one being a 175gb archive with many files included, which would take many tens of seconds until it finished parsing the file list due to the code linked above.

We thus added some options:

{
  "fileIdx": 1,
  "fileMustInclude": ["hello world", "/dexter/i"]
}

These were added so we can stop parsing when we find the file that we require, and also to skip adding the file chunks for all files we do not require. (you can see the changes here: https://github.com/Stremio/rar-stream/blob/ee358ffe86b7060e554aa326a14d15e6f0cb739d/src/rar-files-package.ts#L55-L95 )

As far as we can tell the traversal to get the file list needs to be sequential and there is no shortcut to this method.

We were thinking that one way that we could continue using this module (and not a fork of it) would be to possibly simplify these options so they could be useful for more users, such as:

{
  "maxFiles": 1,
  "filter": (filename, fileIdx) => {
    return filename.includes('hello world')
  }
}

Otherwise, you did absolutely amazing work! I kept stumbling over your module through the years and kept disregarding it because I couldn't wrap my mind around what missing decompression means for RAR archives, and it randomly came to me recently that we don't need decompression at all for our usecase.

Our experiments lead to making this module: https://github.com/stremio/rar-http

It adds support for URLs and uses an HTTP API. I believe that support for URLs could also be added directly to your module, but if we want to keep the spirit of a self-contained (no dependencies) module, it may be hassle to use the http / https modules directly for this task.

We're mostly curious about your opinion on all of this, and if you would even accept PRs for such changes.

Thank you for building rar-stream!

Will there be support for decompression?

I am using cloud commander. A nice way to manage files in web browser. Unfortunately I can't decompress rar files (coderaiser/cloudcmd#222). The author of the app mentioned that it uses rar-stream and rar-steam would not support it.

Will there be support for decompression? Really looking forward to see this feature coming.

Documentation

Is there some basic documentation anywhere?

Does it load the extracted files into memory?

Does it work with partial archives (e.g. .R00 is complete, R01 is not)?

Thank you