future_map raise an error when .f contains a call to a python function imported via the reticulate package

[abdellah19jan]

future_map raise an error when .f contains a call to a python function imported via the reticulate package

[DavisVaughan]

Python functions are a type of non exportable objects that future (which underlies furrr) has no way to deal with. See this part of the error:

Detected a non-exportable reference (‘externalptr’) in one of the globals (‘py_f’ of class ‘python.builtin.function’) used in the future expression.

https://future.futureverse.org/articles/future-4-non-exportable-objects.html

There is even a specific section about reticulate
https://future.futureverse.org/articles/future-4-non-exportable-objects.html#package-reticulate

Something like this might work, but in general I think you are probably going to have a tough time combining R parallelism with reticulate

library(furrr)

plan(multisession, workers = 7)

.f <- function(x) {
  library(reticulate)
  
  use_python(paste0("C:/Users/", Sys.getenv("USERNAME"), "/Anaconda3/python.exe"),
             required = TRUE
  )
  
  source_python("test.py")
  
  py_f(x)
}

future_map(c(1, 2), .f)