Clarification docs: Max 8 minutes on Android

[eivihnd]

Clarification docs: Max 8 minutes on Android

[781flyingdutchman]

At the moment, yes because of the 10 minute limit on Android background workers. However, there is a workaround, see issue #3 and I just haven't gotten to that. May include in V5.0 which also adds pause/resume and should be ready soon

[eivihnd]

Thank you for the clarification, and the work you do on developing this package! Would be greatly appreciated if you manage to include it in V5.

[781flyingdutchman]

Hi, I looked into this. The issue is that in order to have Android allow the task to run for more than 10 minutes, it needs to be set as a foreground task. But that requires a notification to the user, and that requires internationalization (e.g. what text to show) and permissions. The complexity this adds is really not what I want to do, because for the vast majority of use cases 10 minutes is plenty.

One way to get around this for your use case is to break the download up into pieces (assuming the server allows this - check if the "Accept-Ranges" header is present in the response, along with the "Content-Length"). If both are, you can create multiple download tasks in a batch, and for each set a header like this: "Range": "bytes=12345-56789". This will download that range only for that task, and store it in the file you specify. You can then stitch these files together once all are downloaded. It sounds more complicated than it is, but it's also not great, sorry!

If you are comfortable sending me a url with such a big file, I can try to play with this and see if I can implement the above approach as a utility function.

[781flyingdutchman]

@eivihnd good news! Version 5.1.0 now supports arbitrarily long downloads, provided you set the allowPause property of your DownoadTask to true. Under the hood, what happens on Android is that just before the download gets killed by the Android OS (after 9 minutes) we pause the task and immediately resume it on a new worker. By chaining these workers you eventually download the entire file. The only disadvantage is that you will get a .paused -> .enqueued -> .running status update every time this happens, but if you simply wait for a final state (like .complete or failed) then you won't really notice.

I can't really test this feature well with a real file (so I simulate it by forcing timeout after a few milliseconds), so would you mind testing this and seeing if it works for your app? I'll close the bug, but would appreciate your reply.

[eivihnd]

Amazing work!
I will test and give you feedback, hopefully later today

[rekire]

This is an interesting trick to be honest. Are you sure that this time windows are really fixed? I could imagine that there are exceptions where this could be shorter.

[eivihnd]

Initial test using your example app with insignificant changes (apart from allowPause = true), is successful with a large file (~ 1GB) downloaded in approximately 45 minutes (internet speed capped at 2Mbps).

Console log

### Output from console

I/TaskWorker(27114): Starting task with taskId 2683605416
D/TaskWorker(27114): Download for taskId 2683605416
D/TrafficStats(27114): tagSocket(112) with statsTag=0xffffffff, statsUid=-1
W/nloader_example(27114): Reducing the number of considered missed Gc histogram windows from 385 to 100
I/TaskWorker(27114): Task 2683605416 paused due to timeout, will resume in 1 second
I/BackgroundDownloader(27114): Starting task with id 2683605416
I/WM-WorkerWrapper(27114): Worker result SUCCESS for Work [ id=07344d5a-a75f-4cd1-a948-0fc67619eb92, tags={ taskId=2683605416, com.bbflight.background_downloader.TaskWorker, group=default, BackgroundDownloader } ]
I/TaskWorker(27114): Resuming task with taskId 2683605416
D/TaskWorker(27114): Download for taskId 2683605416
D/TrafficStats(27114): tagSocket(112) with statsTag=0xffffffff, statsUid=-1
D/TaskWorker(27114): Resume start=127157653, end=630313012 of total=630313013 bytes, tempFile = 127157653 bytes
I/TaskWorker(27114): Task 2683605416 paused due to timeout, will resume in 1 second
I/BackgroundDownloader(27114): Starting task with id 2683605416
I/WM-WorkerWrapper(27114): Worker result SUCCESS for Work [ id=ecf4782f-5088-4b6f-84dd-76ffb201b0bc, tags={ taskId=2683605416, com.bbflight.background_downloader.TaskWorker, group=default, BackgroundDownloader } ]
I/TaskWorker(27114): Resuming task with taskId 2683605416
D/TaskWorker(27114): Download for taskId 2683605416
D/TrafficStats(27114): tagSocket(112) with statsTag=0xffffffff, statsUid=-1
D/TaskWorker(27114): Resume start=255633116, end=630313012 of total=630313013 bytes, tempFile = 255633116 bytes
I/TaskWorker(27114): Task 2683605416 paused due to timeout, will resume in 1 second
I/BackgroundDownloader(27114): Starting task with id 2683605416
I/WM-WorkerWrapper(27114): Worker result SUCCESS for Work [ id=fa1ac020-74fb-471f-9cda-7a3fad1cef96, tags={ taskId=2683605416, com.bbflight.background_downloader.TaskWorker, group=default, BackgroundDownloader } ]
I/TaskWorker(27114): Resuming task with taskId 2683605416
D/TaskWorker(27114): Download for taskId 2683605416

...

I/TaskWorker(27114): Task 2683605416 paused due to timeout, will resume in 1 second
I/BackgroundDownloader(27114): Starting task with id 2683605416
I/WM-WorkerWrapper(27114): Worker result SUCCESS for Work [ id=2f80f6f2-bc92-461a-926b-8cd774a36ae3, tags={ taskId=2683605416, com.bbflight.background_downloader.TaskWorker, group=default, BackgroundDownloader } ]
I/TaskWorker(27114): Resuming task with taskId 2683605416
D/TaskWorker(27114): Download for taskId 2683605416
D/TrafficStats(27114): tagSocket(112) with statsTag=0xffffffff, statsUid=-1
D/TaskWorker(27114): Resume start=512632365, end=630313012 of total=630313013 bytes, tempFile = 512632365 bytes
I/ViewRootImpl@e27cdb9[MainActivity](27114): ViewPostIme pointer 0
I/ViewRootImpl@e27cdb9[MainActivity](27114): ViewPostIme pointer 1
I/TaskWorker(27114): Successfully downloaded taskId 2683605416 to /data/user/0/com.bbflight.background_downloader_example/app_flutter/my/directory/zipfile.zip
I/WM-WorkerWrapper(27114): Worker result SUCCESS for Work [ id=90661cc2-8cd6-48a6-b89d-b6755faee1ef, tags={ taskId=2683605416, com.bbflight.background_downloader.TaskWorker, group=default, BackgroundDownloader } ]

Will do more extensive tests in the coming weeks.
Thank you very much!

[781flyingdutchman]

HI, you can now also configure the downloader to run Android tasks as a foreground service. It is my understanding that that also eliminates the 9 minute timeout, so you should be able to download the big file regularly (i.e. without intermittent pause/resume). There are some drawbacks to using foreground service (and you do need a notification - it's not optional) but this is an alternative.

[IKTANIM]

For me it works once. But for all the other it is getting an error and download failed.

/TaskWorker(14544): Task 2216092462 paused due to timeout, will resume in 1 second
I/BackgroundDownloader(14544): Enqueuing task with id 2216092462
I/WM-WorkerWrapper(14544): Worker result SUCCESS for Work [ id=13d1165e-1d1f-4aff-94b1-c03a723e0719, tags={ com.bbflight.background_downloader.DownloadTaskWorker, BackgroundDownloader, taskId=2216092462, group=default } ]
I/TaskWorker(14544): Resuming task with taskId 2216092462
D/TaskWorker(14544): Resume start=636561961, end=1808794203 of total=1808794204 bytes, tempFile = 636561961 bytes
I/TaskWorker(14544): Cannot resume: ETag is not identical, or is weak
I/flutter (14544): x12status: TaskResumeException: Cannot resume: ETag is not identical, or is weak
I/flutter (14544): x12exception: TaskResumeException: Cannot resume: ETag is not identical, or is weak
I/flutter (14544): x12eresponseBody: null
I/WM-WorkerWrapper(14544): Work [ id=30aa38ac-5fbd-4d13-9b3b-94970c536d12, tags={ com.bbflight.background_downloader.DownloadTaskWorker, BackgroundDownloader, taskId=2216092462, group=default } ] was cancelled
I/WM-WorkerWrapper(14544): java.util.concurrent.CancellationException: Task was cancelled.
I/WM-WorkerWrapper(14544): 	at androidx.work.impl.utils.futures.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:1184)
I/WM-WorkerWrapper(14544): 	at androidx.work.impl.utils.futures.AbstractFuture.getDoneValue(AbstractFuture.java:514)
I/WM-WorkerWrapper(14544): 	at androidx.work.impl.utils.futures.AbstractFuture.get(AbstractFuture.java:475)

[781flyingdutchman]

The error Cannot resume: ETag is not identical, or is weak says it all: for this to work, your server must provide a 'strong' ETag and it needs to be the same (for the same file). See here

[IKTANIM]

But the same file is being downloaded using dio.

dio.download(
  url,
  filePath,
  onReceiveProgress: (received, total) {
    final progress = (received / total * 100).floor();
    print('Download progress: $progress% $received -- $total');
    // showDownloadProgress(received, total);
    // You can update the progress if needed.
  },
).timeout(Duration(days: 10));

[781flyingdutchman]

If you run into the 9 minute limit, then the download is paused and resumed a few seconds later. For that to work, the server needs to support pause/resume (which includes providing a strong ETag to confirm that the file is the same as the one you started downloading before the pause). In your case, the server does not do that (I don't know why).
The reason this works fine using Dio is that Dio doesn't pause/resume the download.

You should probably configure the downloader to run Android downloads as a foreground service (this eliminates the 9 minute limit but has some other prerequisites, like needing a notification). See the docs under "Configuration" for details on how to do that.

[IKTANIM]

I am using amzon s3 server. I am trying to download from this link.
https://reedevents-pub.s3.ap-southeast-2.amazonaws.com/deakin/videos/S240213DK001-4b7d3dbb-caf8-11ee-bd56-020fb18c1ee8.mp4