robbederks/downzip

NPM library to enable client-side code to stream potentially large files into a zipped download

DownZip

The package.json description says it all: "Library to enable client-side code to stream potentially large files into a zipped download"

Features

• Client-side generation of ZIP files from supplied single-file download URLs
• Support for ZIP64 (ZIP files bigger than 4GB)
• Everything is streamed, no data has to be held in RAM
• No compression is applied, so the CPU load is tiny

Installation

1. Install npm package: npm install downzip
2. Make sure the service worker is copied to your output directory. For example, setup copy-webpack-plugin to put the service worker in your output directory:

// Add copy rule for downzip service worker
new CopyPlugin([
    {
        from: 'node_modules/downzip/dist/downzip-sw.js',
        to: '.'
    }
])

Example usage

import DownZip from 'downzip'

// Setup downzip object
const downZip = new DownZip()
await downZip.register()

// Initialize download
const downloadId = "aaaabbbb"
const zipFileName = "downzip-file"
const files = [
    {
        name: 'picture1.jpg', 
        downloadUrl: 'http://your-download-url.com/picture1.jpg',
        size: 1234      // In bytes
    }, 
    {
        name: 'picture2.jpg', 
        downloadUrl: 'http://your-download-url.com/picture2.jpg',
        size: 4567      // In bytes
    }
]
const downloadUrl = await downZip.downzip(
    downloadId,
    zipFileName,
    files
)

// Start download when user clicks the link
<a href={downloadUrl}>Click to start downloading!</a>

service-worker-loader options

Can pass mapScriptUrl function to the register method. That function gets used by service-worker-loader. docs

    const mapScriptUrl = scriptUrl => scriptUrl.replace('localhost', "127.0.0.1")

   // Setup downzip object
    const downZip = new DownZip()
    await downZip.register({ mapScriptUrl })

TODO

All improvements are welcome, but the main things that need to be improved at the moment are:

• Automated Testing
• Only send the keep-alive service worker message when there is a download queued up / in progress
• Find an easier way to install this package / service worker in projects

Please submit pull requests, I'm more than happy to merge in improvements!

downzip's Issues

No luck getting this to work with svelte-kit

I used rollup-plugin-copy as in the readme example.

(vite.config.js)

import copy from 'rollup-plugin-copy';
const config: UserConfig = {
  plugins: [
    copy({
      targets: [
        {
          src: 'node_modules/downzip/dist/downzip-sw.js',
          dest: './static',
        },
      ],
    }),
    sveltekit(),
  ],
};

I attempted to load and use the module like this:

(svelte component)

  // @ts-ignore
  import DownZip from 'downzip';
  import { onMount } from 'svelte';

  let downZip: any;
  onMount(async () => {
    // Setup downzip object
    downZip = new DownZip();
    await downZip.register();
  });

  let downloadId: number = 0;
  async function downloadAll( fileArray: files[] ) {
      // Initialize download
      let zipFileName = `downzip.${content.id}.zip`;
      const files = fileArray.map((f) => {
        return {
          name: f.name,
          downloadUrl: f.file_url,
          size: f.size_bytes,
        };
      });
      console.log(downZip);
      const url = await downZip.downzip(
        `downZip-${downloadId}`,
        zipFileName,
        files
      );
      downloadURL(url, zipFileName);
      downloadId += 1;
  }

File Size: If size is not available pre-fetch

is there a way for us to not have to pass in size if this is dynamically pulling files? otherwise, shall 0 bytes be passed in?

Move to webpack 5

It looks like webpack 5 has a built-in way to load service workers, which would allow us to get rid of service-worker-loader as a dependency. This package is quite stale anyways, so this would be a nice improvement!

Download finishes midway for 6 gb file

I am receving the issue with download finishing midway for 6 gb zip. I am using chrome on mac. Some of the first files inside the downloaded zip open fine, while the other half do not open, and each weigh 400 bytes only. All files are png and open fine through url.

Triggering the link in the example

Hi,

In your example code, how does the link get triggered? I've used this line from the example literally:

<a href={downloadUrl}>Click to start downloading!</a>

But obviously, clicking the link just sends the browser to /{downloadUrl}. I thought maybe I should set it to # and use onClick to trigger the JavaScript function, but it's not wrapped in a function so I'm just sure what I would change it to.

Sorry if I'm missing something totally obvious. I'm testing this all in a single .html file with the files from your dist folder copied into the same folder:

<html>
<head>
    <script src="main.js"></script>
    <script src="downzip-sw.js"></script>

</head>
<body>
    <p>We'll be exporting attachments here.</p>
    <a href={downloadUrl}>Click to start downloading!</a>
</body>
<script>
    import DownZip from 'downzip'

    // Setup downzip object
    const downZip = new DownZip()

    // Initialize download
    const downloadId = "aaaabbbb"
    const zipFileName = "downzip-file"
    const files = [
        {
            name: 'picture1.jpg'
            downloadUrl: 'http://your-download-url.com/picture1.jpg'
            size: 1234      // In bytes
        },
        {
            name: 'picture2.jpg'
            downloadUrl: 'http://your-download-url.com/picture2.jpg'
            size: 4567      // In bytes
        }
    ]
    const downloadUrl = await downZip.downzip(
        downloadId,
        zipFileName,
        files
    )
</script>
</html>

My background is just vanilla JavaScript, I'm trying to integrate this into a Django app I'm writing. Any pointers would be appreciated.

Thanks!

Ability to interrupt a download?

Hey there, first of all, thanks for making this library available 🙏

I have been using it for some time and notice that when a large zip download is started, it seems that it cannot stop?

• If I close the tab and open the app in another tab, and I see the files continuing to be downloaded in the network tab.
• Simply interrupting the download in Google Chrome does not stop the fetching either

Is there a way to:

1. Stop the download automatically under some circumstances (user cancelled download)
2. Stop the download programmatically (calling a method or sending a message to the worker?)

Feature request: Directories

It'd be useful to be able to create directories inside the generated zip file and place files inside these sub-directories.

Any idea how much work that would entail?

It seems you're already using jszip so it should be relatively straightforward. EDIT: jszip is only used in tests, in such case I recommend moving it from dependencies into devDependencies.

is this based on jzip

Hi.

Thanks for this library, trying to get it going ...

this is based on jszip correct? which means it does support large (more than 4gb) zips?

I set up the service worker using

await navigator.serviceWorker.register('downzip-sw.js');
no error is thrown after this line.

but i keep getting the error when i try to download the file.

main.js?9aaa:1 [DownZip-Main] [DownZip] No service worker registered!

Does this s support ie11? Does it download the remote url first and then create a zip on the client? or does it stream download and create the zip at the same time? which means the client isn't waiting for a progress.

Documentation file array format

In the documentation it says to organize the file array in the following format

const files = [
{
name: 'picture1.jpg'
downloadUrl: 'http://your-download-url.com/picture1.jpg'
size: 1234 // In bytes
},
{
name: 'picture2.jpg'
downloadUrl: 'http://your-download-url.com/picture2.jpg'
size: 4567 // In bytes
}
]

however in the downzip.js file in node_modules their is a comment that says
// Files array is in the following format: [{name: '', url: ''}, ...]

does it matter if its url or downloadUrl ?, does the order matter?

config object for constructor

one config I can think of is for an app that has routing, serviceWorker scope should be defined at the base level

"no file" failures when using with svelte-kit adapter-node

relates to #39

I have managed to get files to download with the svelte-kit dev server (vite dev) but I get a "no file" failure when I use the same code on my production deployments...

The url comes back from downzip('process id', 'zip name', filedata[]) as "downzip/download-0"

hash service worker minified name

Can you name the minified sw.js file something like downzip.1.0.2.min.js? this way it won't be cached in people's browsers in case the lib is updated on next deploy

make sure serviceWorker is registered before running

since this lib handles registering the service worker, should the service worker be manually checked or maybe async/await the construction of the DownZip object?

Remove console logs from prod

I'm getting console logs for service worker successfully registering on app load, there may be more

mapScriptUrl not called in the latest build

When we swapped back over to your recently updated npm package, it wasn't working for us. Seem like the issue is mapScriptUrl never gets called by service-worker-loader.

I think another re-build of the package may solve it?

Support for direct download (no-zip) of individual URLs

Many thanks for creating this tool, which should help our users download multiple files from our website.

I have a somewhat strange request though: can downzip be extended to support downloading a single file without creating a zip? The reason for this is that we store our data on another host so chrome disallows direct download from the URL. Whereas downzip allows us to download multiple files from the host, it would be convenient for our users to download a single file using the same mechanism. I suppose the logic could be like

1. If files contains multiple files, use .zip
2. if files contains a single file and zipFileName is unspecified or does not end with .zip, download in original format.

Can you help me to use a downzip, please?

Hi @robbederks, I need help. I want to use your downzip library. You can indicate the steps, I cannot configure it.

I have a js folder where I saved the js files. What files do I have to download? In the js part, what code do I have to put, I just didn't make it work.

Thank you!

Resource interpreted as Document but transferred with MIM type application/octet-stream

In chrome, I get the warning above. The resulting zip is around 15 bytes. In the Chrome Network developer tool I see the downloading of the zip contents (~9MB). Any tips or clues on how to solve this?

Memory leak in service-worker

hi,

I plan on using this script to download very large files, so I set up a quick test-project to try it out.

https://github.com/mindplay-dk/downzip/tree/original-downzip

in the files folder, I have 10 files of about 0.46 GB each, and as you can see, I'm zipping and downloading them all - the idea here was just to see if this would work for over 4 GB of files, which it actually does. 🙂👍

it appears there's a memory issue though - while I'm downloading 4.6GB of files, the service worker appears to eat up 4.6GB of RAM, as you can see in this screen recording:

(for the record, the progress events you see in this recording is from this fork, which has a few extra features - I did test with your original package as well, which is what I'm using in the repo I linked to above, and it does have the same issue, I just didn't bother with a second screen recording.)

also, as you can see, it does not appear to really be streaming (?) so much as buffering the whole thing in the service worker - it appears to work up to the full 4.6GB of memory usage before the download really begins. this is probably a related issue. (and this is true of both your version and that fork as well.)

what's worse, memory appears to remain allocated after the download finishes - sometimes all of it, sometimes about half, sometimes for several minutes after closing the window. I suppose this could just be how Chrome operates service-workers? no idea.

I'm using latest Chrome on Win 11 Pro: Version 120.0.6099.71 (Official Build) (64-bit).

this all seemed rather odd, so I decided to give Firefox a go as well.

in Firefox, the downloads begin immediately - there's no delay, like there is in Chrome.

however, the downloads do not complete - they appear to randomly hang after 5-10 seconds of downloading:

similarly, memory usage does appear to climb while downloading, to about the same as amount of memory as the amount of the download that completes before it hangs - so the likely explanation is Firefox hitting the service-worker with some sort of memory usage limit designed to keep service-workers from eating all memory? however, there's no indication of any error in the debugger - and the downloads stall, forever, they're never canceled.

again, this is with Firefox on Win 11 Pro: 120.0.1 (64-bit).

just for completeness, I also gave Edge a shot - as you might expect, the experience is the same as in Chrome.

returning to Chrome, I also attempted to profile memory usage - oddly, if the memory profiler is running, memory will get freed immediately after the download, much sooner than without devtools... as you can see though, it is clearly allocating all the memory while the service-worker is downloading, before the host browser actually begins downloading the zip stream:

that's all I have right now - I will get back to you if I uncover anymore clues.

Getting error while integrating this in React project

Hi,

I am trying to integrate this library in a react project but not able to make it work .

I am getting below error.

The script has an unsupported MIME type ('text/html'). [DownZip-Main] [DownZip] Service workers not loaded: [DownZip-Main] [DownZip] No service worker registered! Uncaught (in promise) TypeError: Cannot read property 'postMessage' of null
Is there any guideline or sample project on how to use this in react project.