Comments (5)
DavisVaughan
commented on July 29, 2024
1
That error comes from base R, not from furrr.
This vignette explains in more detail
https://future.futureverse.org/articles/future-4-non-exportable-objects.html
You could try setting options(future.globals.onReference = "error") as recommended here to try and figure out where it comes from https://future.futureverse.org/articles/future-4-non-exportable-objects.html#protect-against-non-exportable-objects
from furrr.
DavisVaughan
commented on July 29, 2024
1
I think you can set options(future.debug = TRUE) to get a very detailed log
from furrr.
twest820
commented on July 29, 2024
Thanks! Unfortunately, options(future.globals.onReference = "error") seems to misunderstand other code and errors out on something which doesn't look like it should be a problem.
Error in (function (.x, .f, ..., .progress = FALSE) : ℹ In index: 1.
Caused by error:
! Detected a non-exportable reference (‘externalptr’) in the value (of class ‘LAS’) of the resolved futureThis appears to be triggered by the LAS object returned from a call to lidR::clip_roi(). However, those objects are manipulated only at furrr function body scope, are held only in memory, and thus never need to be transferred between workers (or written to disk). So there shouldn't be need to export them.
from furrr.
DavisVaughan
commented on July 29, 2024
I'd have to see a minimal example. furrr and future seem to be trying to export that object to your workers
from furrr.
twest820
commented on July 29, 2024
I'd have to see a minimal example.
Only a one line difference from #259.
library(dplyr)
library(furrr)
library(sf)
library(terra)
plan(multisession, workers = 16)
simpleFeatureCollection = st_read("simpleFeatureCollection.gpkg")
mediumSizeRaster = rast("twoGBraster.tif") # global in this case
with_progress({
progressBar = progressor(steps = nrow(simpleFeatureCollection))
future_map(simpleFeatureCollection$ID, function(polygonID)
{
regionOfInterestPolygon = (simpleFeatureCollection %>% filter(ID == polygonID))[1]
rasterRegionOfInterest = crop(mediumSizeRaster, regionOfInterestPolygon)
<do computationally intensive things>
progressBar(<update message>)
})
})I think you can set
options(future.debug = TRUE)to get a very detailed log
Indeed! Unfortunately I'm not spotting anything illuminating. Or seeing anything in the other future options which would write out the failed FutureResult to get at the call stack and maybe have some minimal hint at which symbol is NULL. Seems like this has been, at least partially, a known limitation of future for some time (HenrikBengtsson/future#478). I guess the workaround would to be write against GDAL in C++ with the task parallel library to get a normal amount of diagnostic information, though maybe having this issue will help a little with prioritizing work over in future.
[11:03:35.758] MultisessionFuture started
[11:03:35.758] - Launch lazy future ... done
[11:03:35.759] run() for ‘MultisessionFuture’ ... done
[11:03:35.759] resolve() on list ...
[11:03:35.759] recursive: 0
[11:03:35.759] length: 2
[11:03:35.759]
[11:03:35.999] receiveMessageFromWorker() for ClusterFuture ...
[11:03:35.999] - Validating connection of MultisessionFuture
[11:03:36.000] - received message: FutureResult
[11:03:36.000] - Received FutureResult
[11:03:36.000] - Erased future from FutureRegistry
[11:03:36.000] result() for ClusterFuture ...
[11:03:36.000] - result already collected: FutureResult
[11:03:36.001] result() for ClusterFuture ... done
[11:03:36.001] signalConditions() ...
[11:03:36.001] - include = ‘immediateCondition’
[11:03:36.001] - exclude =
[11:03:36.001] - resignal = FALSE
[11:03:36.001] - Number of conditions: 1
[11:03:36.001] signalConditions() ... done
[11:03:36.002] receiveMessageFromWorker() for ClusterFuture ... done
[11:03:36.002] Future #2
[11:03:36.002] result() for ClusterFuture ...
[11:03:36.002] - result already collected: FutureResult
[11:03:36.002] result() for ClusterFuture ... done
[11:03:36.002] result() for ClusterFuture ...
[11:03:36.002] - result already collected: FutureResult
[11:03:36.002] result() for ClusterFuture ... done
[11:03:36.002] signalConditions() ...
[11:03:36.003] - include = ‘immediateCondition’
[11:03:36.003] - exclude =
[11:03:36.003] - resignal = FALSE
[11:03:36.003] - Number of conditions: 1
[11:03:36.003] signalConditions() ... done
[11:03:36.003] signalConditionsASAP(MultisessionFuture, pos=2) ...
[11:03:36.003] - nx: 2
[11:03:36.003] - relay: TRUE
[11:03:36.004] - stdout: TRUE
[11:03:36.004] - signal: TRUE
[11:03:36.004] - resignal: FALSE
[11:03:36.004] - force: TRUE
[11:03:36.004] - relayed: [n=2] FALSE, FALSE
[11:03:36.004] - queued futures: [n=2] FALSE, FALSE
[11:03:36.004] - until=1
[11:03:36.004] - relaying element #1
[11:03:36.004] - relayed: [n=2] FALSE, FALSE
[11:03:36.005] - queued futures: [n=2] FALSE, TRUE
[11:03:36.005] signalConditionsASAP(NULL, pos=2) ... done
[11:03:36.005] length: 1 (resolved future 2)
[11:03:46.083] receiveMessageFromWorker() for ClusterFuture ...
[11:03:46.083] - Validating connection of MultisessionFuture
[11:03:46.084] - received message: FutureResult
[11:03:46.084] - Received FutureResult
[11:03:46.085] - Erased future from FutureRegistry
[11:03:46.085] result() for ClusterFuture ...
[11:03:46.085] - result already collected: FutureResult
[11:03:46.085] result() for ClusterFuture ... done
[11:03:46.085] signalConditions() ...
[11:03:46.085] - include = ‘immediateCondition’
[11:03:46.085] - exclude =
[11:03:46.085] - resignal = FALSE
[11:03:46.086] - Number of conditions: 1
[11:03:46.086] signalConditions() ... done
[11:03:46.086] receiveMessageFromWorker() for ClusterFuture ... done
[11:03:46.086] Future #1
[11:03:46.086] result() for ClusterFuture ...
[11:03:46.086] - result already collected: FutureResult
[11:03:46.086] result() for ClusterFuture ... done
[11:03:46.086] result() for ClusterFuture ...
[11:03:46.086] - result already collected: FutureResult
[11:03:46.086] result() for ClusterFuture ... done
[11:03:46.087] signalConditions() ...
[11:03:46.087] - include = ‘immediateCondition’
[11:03:46.087] - exclude =
[11:03:46.087] - resignal = FALSE
[11:03:46.087] - Number of conditions: 1
[11:03:46.087] signalConditions() ... done
[11:03:46.088] signalConditionsASAP(MultisessionFuture, pos=1) ...
[11:03:46.088] - nx: 2
[11:03:46.088] - relay: TRUE
[11:03:46.088] - stdout: TRUE
[11:03:46.088] - signal: TRUE
[11:03:46.088] - resignal: FALSE
[11:03:46.088] - force: TRUE
[11:03:46.088] - relayed: [n=2] FALSE, FALSE
[11:03:46.089] - queued futures: [n=2] FALSE, TRUE
[11:03:46.089] - until=1
[11:03:46.089] - relaying element #1
[11:03:46.089] result() for ClusterFuture ...
[11:03:46.089] - result already collected: FutureResult
[11:03:46.089] result() for ClusterFuture ... done
[11:03:46.089] result() for ClusterFuture ...
[11:03:46.089] - result already collected: FutureResult
[11:03:46.089] result() for ClusterFuture ... done
[11:03:46.089] signalConditions() ...
[11:03:46.090] - include = ‘immediateCondition’
[11:03:46.090] - exclude =
[11:03:46.090] - resignal = FALSE
[11:03:46.090] - Number of conditions: 2
[11:03:46.090] signalConditions() ... done
[11:03:46.090] result() for ClusterFuture ...
[11:03:46.090] - result already collected: FutureResult
[11:03:46.090] result() for ClusterFuture ... done
[11:03:46.090] signalConditions() ...
[11:03:46.091] - include = ‘immediateCondition’
[11:03:46.091] - exclude =
[11:03:46.091] - resignal = FALSE
[11:03:46.091] - Number of conditions: 2
[11:03:46.091] signalConditions() ... done
[11:03:46.091] result() for ClusterFuture ...
[11:03:46.091] - result already collected: FutureResult
[11:03:46.091] result() for ClusterFuture ... done
[11:03:46.091] signalConditions() ...
[11:03:46.092] - include = ‘condition’
[11:03:46.092] - exclude = ‘immediateCondition’
[11:03:46.092] - resignal = TRUE
[11:03:46.092] - Number of conditions: 2
[11:03:46.092] - Condition #1: ‘RngFutureWarning’, ‘FutureWarning’, ‘warning’, ‘RngFutureCondition’, ‘FutureCondition’, ‘condition’
[11:03:46.092] - Condition #2: ‘purrr_error_indexed’, ‘rlang_error’, ‘error’, ‘condition’
Error in (function (.x, .f, ..., .progress = FALSE) :
ℹ In index: 1.
Caused by error in `.External()`:
! NULL value passed as symbol address
[11:03:46.120] signalConditions() ... done
[11:03:46.120] - relayed: [n=2] FALSE, FALSE
[11:03:46.121] - queued futures: [n=2] TRUE, TRUE
[11:03:46.121] signalConditionsASAP(MultisessionFuture, pos=1) ... done
from furrr.
Related Issues (20)
- Error indices from purrr() are misleading in furrr()
- Getting the same random result across `purrr::map()` and `furrr::future_map()` HOT 3
- `furrr` much slower than `purrr` (on Windows) HOT 2
- Export a wrapper around purrr `map()` to workers rather than `map()` itself
- Setting `seed = TRUE` / `.progress=TRUE` globally in the R session HOT 1
- Carrier HOT 2
- mgcv::gam(~s(pc = object not found)) when future::plan(multisession, workers > 1) HOT 1
- when large matrix or df split into list, does furrr workers only access list element? HOT 1
- terra::rast() doesn't return a variable with workers > 1 HOT 3
- Understanding memory usage and performance of `furrr::future_apply` HOT 5
- Implement map_vec
- Use of variables in glue() inside furrr loops HOT 1
- Progress bar showing 100% even if not completed
- Error when plotting a {sf} object
- Identical RNG state for each task, despite setting different seeds in different tasks HOT 10
- How to pass to .f, character vectors in .x that reference global values?
- `future_map` not obviously faster than `map` in simple linear regression setting HOT 1
- Multiple calls of future_map() within a single plan() or script result in massive slowdown.
- furrr_options documentation doesn't say what to do with it
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from furrr.