Ability to use FileDownloader on a child isolate

[cal-g]

Ability to use FileDownloader on a child isolate

[781flyingdutchman]

TL;DR: The new approach in Flutter 3.7 is only for messages from Fluttter to a plugin, and does not yet solve for messages from a plugin to Flutter. Therefore, the plugin cannot be used within an isolate on native (i.e mobile) platforms.

=====

Hi, I've tried to use the background_downloader on an isolate using the new Flutter 3.7 feature and it works fine. I stayed close to the example code listed in the article you mentioned. Here is the code:

import 'dart:async';
import 'dart:collection';
import 'dart:isolate';

import 'package:background_downloader/background_downloader.dart';
import 'package:flutter/foundation.dart';
import 'package:flutter/services.dart';

enum _Codes { init, ack, download, result }

/// A command sent between main and download isolate.
class _Command {
  const _Command(this.code, {this.arg});

  final _Codes code;
  final Object? arg;
}

class MainDownloader {
  MainDownloader._(this._isolate);

  final Isolate _isolate;
  late final SendPort _sendPort;

  // Completers are stored in a queue so multiple commands can be queued up and
  // handled serially.
  final Queue<Completer<void>> _completers = Queue<Completer<void>>();

  // Result is in a completer
  var _resultCompleter = Completer<TaskStatusUpdate>();

  /// Start the downloader Isolate
  static Future<MainDownloader> start() async {
    final ReceivePort receivePort = ReceivePort();
    final Isolate isolate =
        await Isolate.spawn(IsolateDownloader._run, receivePort.sendPort);
    final MainDownloader result = MainDownloader._(isolate);
    Completer<void> completer = Completer<void>();
    result._completers.addFirst(completer);
    receivePort.listen((message) {
      result._handleCommand(message as _Command);
    });
    await completer.future;
    return result;
  }

  /// Loads a file from [url] and returns the [TaskStatusUpdate]
  Future<TaskStatusUpdate> download(String url) {
    // No processing happens on the calling isolate, it gets delegated to the
    // background isolate
    _resultCompleter = Completer();
    _sendPort.send(_Command(_Codes.download, arg: url));
    return _resultCompleter.future;
  }

  /// Handler invoked when a message is received from the port communicating
  /// with the [IsolateDownloader].
  void _handleCommand(_Command command) {
    switch (command.code) {
      case _Codes.init:
        _sendPort = command.arg as SendPort;
        // ----------------------------------------------------------------------
        // Before using platform channels and plugins from background isolates we
        // need to register it with its root isolate. This is achieved by
        // acquiring a [RootIsolateToken] which the background isolate uses to
        // invoke [BackgroundIsolateBinaryMessenger.ensureInitialized].
        // ----------------------------------------------------------------------
        RootIsolateToken rootIsolateToken = RootIsolateToken.instance!;
        _sendPort.send(_Command(_Codes.init, arg: rootIsolateToken));
        break;
      case _Codes.ack:
        _completers.removeLast().complete();
        break;
      case _Codes.result:
        _resultCompleter.complete(command.arg as TaskStatusUpdate);
        break;
      default:
        debugPrint('SimpleDatabase unrecognized command: ${command.code}');
    }
  }

  /// Kills the background isolate.
  void stop() {
    _isolate.kill();
  }
}

/// This is what runs in an isolate
class IsolateDownloader {
  IsolateDownloader(this._sendPort);

  final SendPort _sendPort;

  // ----------------------------------------------------------------------
  // Here the plugin is used from the background isolate.
  // ----------------------------------------------------------------------

  /// The main entrypoint for the background isolate sent to [Isolate.spawn].
  static void _run(SendPort sendPort) {
    ReceivePort receivePort = ReceivePort();
    sendPort.send(_Command(_Codes.init, arg: receivePort.sendPort));
    final IsolateDownloader server = IsolateDownloader(sendPort);
    receivePort.listen((message) async {
      final _Command command = message as _Command;
      await server._handleCommand(command);
    });
  }

  /// Handle the [command] received from the [ReceivePort].
  Future<void> _handleCommand(_Command command) async {
    switch (command.code) {
      case _Codes.init:
        // ----------------------------------------------------------------------
        // The [RootIsolateToken] is required for
        // [BackgroundIsolateBinaryMessenger.ensureInitialized] and must be
        // obtained on the root isolate and passed into the background isolate via
        // a [SendPort].
        // ----------------------------------------------------------------------
        RootIsolateToken rootIsolateToken = command.arg as RootIsolateToken;
        // ----------------------------------------------------------------------
        // [BackgroundIsolateBinaryMessenger.ensureInitialized] for each
        // background isolate that will use plugins. This sets up the
        // [BinaryMessenger] that the Platform Channels will communicate with on
        // the background isolate.
        // ----------------------------------------------------------------------
        BackgroundIsolateBinaryMessenger.ensureInitialized(rootIsolateToken);
        _sendPort.send(const _Command(_Codes.ack, arg: null));
        break;
      case _Codes.download:
        await _doDownload(command.arg as String);
        break;
      default:
        debugPrint('Unrecognized command ${command.code}');
    }
  }

  /// Perform the download with the given url and send the resulting
  /// [TaskStatusUpdate] back to the main isolate via the sendPort.
  Future<void> _doDownload(String url) async {
    debugPrint('Performing download: $url');
    final task = DownloadTask(url: url);
    final result = await FileDownloader().download(task);
    _sendPort.send(_Command(_Codes.result, arg: result));
  }
}

I added it to the example app by calling it from initState after a 2 second delay, just to test:

Future.delayed(const Duration(seconds: 2)).then((_) async {
      debugPrint('Starting download on isolate');
      final downloader = await MainDownloader.start();
      final result = await downloader.download('http://google.com');
      debugPrint('Isolate download result = ${result.status} for taskId ${result.task.taskId}');
      downloader.stop();
    });

On Linux, the output is as expected:

flutter: Starting download on isolate
flutter: Performing download: http://google.com
flutter: Isolate download result = TaskStatus.complete for taskId 3939304884

On Android (and presumably iOS) I get the UI actions are only available on root isolate error pointing to the line in the initialize method of the NativeDownloader that initializes the WidgetsFlutterBinding. Removing that line (and initializing the binding before running the app, which is where it should be, so changing that for a future update) then generates the error Binding has not yet been initialized.. Doing some debugging it is clear the binding has in fact been initialized using the rootIsolateToken approach you followed, and as laid out in the article and example code.

Digging a bit further, the issue is that while they have solved the problem for using Plugins that have a simple call/return from Flutter (like path_provider) they have not yet solved this for plugins that actively message from the plugin to Flutter (i.e. the other way) using a MethodChannel themselves. From the article:

For more information on the implementation, check out the Isolate Platform Channels design doc. This doc also contains proposals for communicating in the opposite direction, which have not been implemented or accepted yet.

So, long story short: the plugin won't work on an isolate (on native platforms) until the Flutter team has solved for the messaging in the other direction.

I'm closing this issue for now (as I can't solve it) but if the situation changes we'll get back to it. Thanks again for your help.

[781flyingdutchman]

@VP2124 the conclusion from our exploration was that background_downloader cannot be used from an isolate. You don't need to try further, it simply doesn't work until the Dart team implements functionality to also enable messages from plugin to Dart on a background isolate.

[dash-vishalparmar]

@781flyingdutchman Okay Thanks for your answer and I have used another package for isolated_download_manager which is works fine for me

[781flyingdutchman]

Hi, thanks for raising this. Unfortunately I have no experience with this new capability in Flutter 3.7, and I won't have time to work on the plugin much over the next couple of weeks, so not sure I can be much help immediately. The error message UI actions are only available on root isolate is a hint, but 1) if the code snippet above is all you do then I really don't see where just initializing the FileDownloader could possibly trigger a UI action, and 2) the logs don't point to anything related to the plugin. Perhaps you can expand the logs at line 10 to see if there is something there? If not, I suspect the issue actually is not related to the plugin - but again, I am not sure and unfortunately can't really dig in right now.

[cal-g]

I will try to look into this further and see if its possible to run it in another isolate and hopefully the downloads still continue the same way when moving the app to the background.
I found that the above error was because of this line but it seems that MethodChannel.setMethodCallHandler is raising another issue when calling in another isolate. I'm a bit new to platform channel communications so I'll try this channel communication in a sample app.

[781flyingdutchman]

The other approach to consider is to send a message from your isolate to the main isolate with the download request, execute it on the main isolate and send the result or even progress back to your isolate via the isolate message structure. It's not as difficult as it sounds and it keeps the downloader plugin running on the main isolate.

…

On Thu, May 4, 2023, 11:26 AM Calvin Gonsalves ***@***.***> wrote: I will try to look into this further and see if its possible to run it in another isolate and hopefully the downloads still continue the same way when moving the app to the background. I found that the above error was because of this line <https://github.com/781flyingdutchman/background_downloader/blob/db512f67193ce1b1d5957a9c505e5ec43b63845f/lib/src/native_downloader.dart#L28> but it seems that MethodChannel.setMethodCallHandler is raising another issue when calling in another isolate. I'm a bit new to platform channel communications so I'll try this channel communication in a sample app. — Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF5VCHYTWHOCLYHWO3NHB53XEPYGTANCNFSM6AAAAAAXTPP2G4> . You are receiving this because you commented.Message ID: ***@***.***>

[781flyingdutchman]

You can take a look at the DesktopDownloader class for inspiration on how to set up two way comms between your isolate and the main one.

[cal-g]

We already have some long running isolates set up for downloading data and sending messages back to the main isolate about the progress for the UI. The DownloadTasks depend on this data downloaded in the child isolate for the correct url so I thought it would be better to also start this FileDownloader.enqueue or FileDownloader.downloadBatch in the same child isolate because I am also tracking the Tasks and didn't want the LocalStore writes to affect the main isolate much. We do a lot of file downloading >3000 files

The other approach to consider is to send a message from your isolate to
the main isolate with the download request, execute it on the main isolate
and send the result or even progress back to your isolate via the isolate
message structure.

Yea, I'm thinking along these lines now. After some testing it doesn't seem to be affecting the main thread.

[781flyingdutchman]

Yes, the downloader is very light on the main thread, and the Localstore is just tiny file read/writes, which run on their own isolate as well, so shouldn't affect the main thread much

[dash-vishalparmar]

@781flyingdutchman I am still getting this error Binding has not yet been initialized. after following all steps as mentioned but I have one question is as below:

Exactly where I can be initialized using the rootIsolateToken I mean where in my project is it in main.dart file or any other please tell me I am stuck here.

[shahmirzali49]

@781flyingdutchman Okay Thanks for your answer and I have used another package for isolated_download_manager which is works fine for me

@cal-g @VP2124 but the package is not working in the background? What did you do?

[cal-g]

@Shahmirzali-Huseynov I do the download in the main isolate.

The other approach to consider is to send a message from your isolate to the main isolate with the download request, execute it on the main isolate and send the result or even progress back to your isolate via the isolate message structure. It's not as difficult as it sounds and it keeps the downloader plugin running on the main isolate.

I communicate back and forth as mentioned earlier in this thread.

Another approach, with the recent changes in tracking downloads in Database I am using my own sqlite DB from the Drift package. The drift works in a separate isolate and I share this drift isolate's port with the isolate (not the main/root isolate but original Isolate where I wished to do the File downloads from) and using Drifts feature of listening to table updates I can avoid sending messages back and forth.

I will need to stick to this until Flutter handles communicating from opposite direction (i.e receiving method calls on the channel) while accessing platform plugins within dart isolate.

[shahmirzali49]

@cal-g thank you for your reply. When you have time can you share some code with us?