Downloads don't complete at all when I trigger lots of downloads (about 120) at the same time

[flying-wizzy]

Downloads don't complete at all when I trigger lots of downloads (about 120) at the same time

[781flyingdutchman]

Hi! Sorry to hear you're having this problem. A few basics:

1. Which version of the package are you using?
2. What platform does this happen on, and what version of that OS?
3. Can you share the code snippet of how you're enqueueing your tasks?

As for fixes, a few ideas:

• Try using `downloadBatch' instead of separate downloads
• If you use enqueue then insert await Future.delayed(const Duration(milliseconds: 200)) after every third enqueue or so. You can look at the implementation of downloadBatch to see how this is done there. Reason is I found that enqueueing without a break sort of chokes the queue.

For what it's worth, I use this package to download >1000 files at once (albeit a bit smaller, more like 2Mb each) and it works without issues so I think we should be able to solve this.

[flying-wizzy]

Thank you for your input, that sounds promising!
I'll try your ideas and will give detailed feedback on your basic questions later today (25.05.) or latest tomorrow.

I'll answer what I can right away:

1. newest version of the package, 7.1.0
2. Android platform, I only tried my Xiaomi Redmi Note 11 (Android 12 SKG1.210908.001) and on a Pixel 6 Emulator with the newest available Android distribution.
3. I'll share the exact snippet ASAP, but it's basically just a for loop that triggers enqueue() for every Task in the List.

[781flyingdutchman]

Thanks. I do think the issue is the for loop without yielding for processing. Here's how it's implemented in downloadBatch (where we also enqueue a list of tasks):

var counter = 0;
    for (final task in tasks) {
      taskFutures.add(_enqueueAndAwait(task,
          taskStatusCallback: taskStatusCallback,
          taskProgressCallback: taskProgressCallback));
      counter++;
      if (counter % 3 == 0) {
        // To prevent blocking the UI we 'yield' for a few ms after every 3
        // tasks we enqueue
        await Future.delayed(const Duration(milliseconds: 50));
      }
    }

You would replace the bit that says taskFutures.add(_enqueueAndAwait with a regular FileDownloader().enqueue.

[flying-wizzy]

I did my testing now, it seems like it was my fault all along, because I am using the library in an edge case at the moment: For testing purposes, I was using the very same download link for all the download tasks, because I don't have the actual download links for the files yet.
Now I have tried some other random but unique 120 download links and the downloads completed as expected. It didn't even need the little yielding for 50 milliseconds to do the trick. I will nevertheless switch to the downloadBatch method in the future, as to not block the UI.

Maybe the operating system in the background is assuming some kind of error, if you try to download the same file too many times and that caused the tasks to get stuck.

So, sorry for bothering you and thank you for helping out this quickly again!

[781flyingdutchman]

No problem, glad it was resolved!