Issue downloading Worldclim

[Uriwolkow]

Issue downloading Worldclim

[dramanica]

Could you please provide me with the command that you are using (so that I know which variables and resolution you are trying to use?). The Worldclim server can be rather temperamental, so it might just be a question of retrying after a while. But if you send the command, I can check if the problem occurs over here as well.

[Uriwolkow]

I was just trying out the code from the basic vignette on my machine.
The line producing the error was:
download_dataset(dataset = "WorldClim_2.1_10m")

[dramanica]

Hmm, this is puzzling. I just run it on my local machine, and tested it with our CI on github, and both are working. So, we need to figure out what is going wrong on your local machine. It might be an issue with the gdal library.
Can you please run the following test code:

  vrt_path <- file.path(tempdir(),"test.vrt")
  tif_files <- list.files(system.file("extdata/CHELSA_bio01", package="pastclim"),
             full.names = TRUE)
  # create the file
 sf::gdal_utils(
    util = "buildvrt",
    source = tif_files,
    destination = vrt_path,
    options = c("-separate","-overwrite")
  )

[Uriwolkow]

I ran this right now, with no errors.
Then tried downloading WorldClim again, but failed.
I should note that it did work with the Beyer2020 dataset.

[dramanica]

Ok, let's try something a bit more through. Could you please run the following set of tests?

library(testthat)
  vrt_path <- file.path(tempdir(),"test.vrt")
  tif_files <- list.files(system.file("extdata/CHELSA_bio01", package="pastclim"),
             full.names = TRUE)
  sf::gdal_utils(
    util = "buildvrt",
    source = tif_files,
    destination = vrt_path,
    options = c("-separate","-overwrite")
  )
  
  
  description <- "band_name_1"
  time_vector <- c(0,-10,-1000)
  expect_true(vrt_set_meta(vrt_path, description, time_vector))
  # check we have the correct description in the file
  vrt_rast <- terra::rast(vrt_path)
  expect_true(identical(names(vrt_rast),paste(description,time_vector, sep="_")))
  vrt_meta <- vrt_get_meta(vrt_path = vrt_path)
  expect_true(identical(vrt_meta$time_bp,time_vector))
  expect_true(identical(vrt_meta$description,description))
  # check that get_time_bp_steps works with a vrt file
  expect_true(identical(
    get_time_bp_steps(dataset="custom", path_to_nc = vrt_path),time_vector))
  
  
  # expect a warning if we try to set the metadata a second time
  expect_warning(vrt_res <- vrt_set_meta(vrt_path, description, time_vector),
               "metadata for pastclim is already present")
  expect_false(vrt_res)
  
  # expect a warning if we pass an incorrect number of times
  sf::gdal_utils(
    util = "buildvrt",
    source = tif_files,
    destination = vrt_path,
    options = c("-separate","-overwrite")
  )
  expect_warning(vrt_res <- vrt_set_meta(vrt_path, description, c(time_vector,4)),
                 "the vrt has a different number of")
  expect_false(vrt_res)

Also, could you please give me the output of:
sf::sf_extSoftVersion()

[Uriwolkow]

getting stuck on this one:

expect_true(vrt_set_meta(vrt_path, description, time_vector))
Error in vrt_set_meta(vrt_path, description, time_vector) : 
  could not find function "vrt_set_meta"
> expect_true(pastclim::vrt_set_meta(vrt_path, description, time_vector))
Error: 'vrt_set_meta' is not an exported object from 'namespace:pastclim'

also, like you asked:

> sf::sf_extSoftVersion()
          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H           PROJ 
      "3.12.1"        "3.8.4"        "9.3.1"         "true"         "true"        "9.3.1"

[dramanica]

Sorry, my fault, I should have exposed the internal pastclim functions. Use this:

library(testthat)
vrt_path <- file.path(tempdir(),"test.vrt")
tif_files <- list.files(system.file("extdata/CHELSA_bio01", package="pastclim"),
                        full.names = TRUE)
sf::gdal_utils(
  util = "buildvrt",
  source = tif_files,
  destination = vrt_path,
  options = c("-separate","-overwrite")
)


description <- "band_name_1"
time_vector <- c(0,-10,-1000)
expect_true(pastclim:::vrt_set_meta(vrt_path, description, time_vector))
# check we have the correct description in the file
vrt_rast <- terra::rast(vrt_path)
expect_true(identical(names(vrt_rast),paste(description,time_vector, sep="_")))
vrt_meta <- pastclim:::vrt_get_meta(vrt_path = vrt_path)
expect_true(identical(vrt_meta$time_bp,time_vector))
expect_true(identical(vrt_meta$description,description))
# check that get_time_bp_steps works with a vrt file
expect_true(identical(
  pastclim:::get_time_bp_steps(dataset="custom", path_to_nc = vrt_path),time_vector))


# expect a warning if we try to set the metadata a second time
expect_warning(vrt_res <- pastclim:::vrt_set_meta(vrt_path, description, time_vector),
               "metadata for pastclim is already present")
expect_false(vrt_res)

# expect a warning if we pass an incorrect number of times
sf::gdal_utils(
  util = "buildvrt",
  source = tif_files,
  destination = vrt_path,
  options = c("-separate","-overwrite")
)
expect_warning(vrt_res <- pastclim:::vrt_set_meta(vrt_path, description, c(time_vector,4)),
               "the vrt has a different number of")
expect_false(vrt_res)

Also, what OS are you on? Thanks.

[Uriwolkow]

Now everything seems to be running ok. I'm using Windows 11

[dramanica]

Great, thanks for trying that. I have just checked our Windows 11 test machine, and it has exactly the same library versions as you, so it is surprising that yours does not work.
If you are willing to do a bit more testing, I could create a debugging version of pastclim for you over the weekend so that we can chase down where things are going wrong on your machine.

[dramanica]

In the mean time, can you please share the output of:
dir(get_data_path())
If it includes "worldclim_2.1", then please also run:
dir(paste0(get_data_path(),"/worldclim_2.1"))
Finally, had you tried using the CRAN version before using dev?

[Uriwolkow]

Sure, we can try that.
here is the output for these last two lines :

dir(get_data_path())
[1] "Beyer2020_annual_vars_v1.2.2.nc" "example_climate_v1.3.0.nc"
[3] "worldclim_2.1" "WorldClim_2.1_10m_bio01_v1.4.0.vrt"
[5] "WorldClim_2.1_10m_bio02_v1.4.0.vrt" "WorldClim_2.1_10m_bio03_v1.4.0.vrt"
[7] "WorldClim_2.1_10m_bio04_v1.4.0.vrt" "WorldClim_2.1_10m_bio05_v1.4.0.vrt"
[9] "WorldClim_2.1_10m_bio06_v1.4.0.vrt" "WorldClim_2.1_10m_bio07_v1.4.0.vrt"
[11] "WorldClim_2.1_10m_bio08_v1.4.0.vrt" "WorldClim_2.1_10m_bio09_v1.4.0.vrt"
[13] "WorldClim_2.1_10m_bio10_v1.4.0.vrt" "WorldClim_2.1_10m_bio11_v1.4.0.vrt"
[15] "WorldClim_2.1_10m_bio12_v1.4.0.vrt" "WorldClim_2.1_10m_bio13_v1.4.0.vrt"
[17] "WorldClim_2.1_10m_bio14_v1.4.0.vrt" "WorldClim_2.1_10m_bio15_v1.4.0.vrt"
[19] "WorldClim_2.1_10m_bio16_v1.4.0.vrt" "WorldClim_2.1_10m_bio17_v1.4.0.vrt"
[21] "WorldClim_2.1_10m_bio18_v1.4.0.vrt" "WorldClim_2.1_10m_bio19_v1.4.0.vrt"
dir(paste0(get_data_path(),"/worldclim_2.1"))
[1] "wc2.1_10m_bio.zip"

I have the 2.0.0 version installedAnd tried using it vefore using 2.1.0.
It printed a different error:

> download_dataset(dataset = "WorldClim_2.1_10m")
Error in curl::curl_download(full_url, destfile = destfile, quiet = FALSE) : 
  Timeout was reached: [] Send failure: Connection was aborted

[dramanica]

OK, I think I know what is going on. There was a crash when donwloading the data the first time, and it failed to clean after itself.
So, let's get rid of any old Worldclim file with:

unlink(paste0(get_data_path(),"/World*"))
unlink(paste0(get_data_path(),"/worldclim_2.1/*"))

Now rerun

dir(get_data_path())

And there should be no WorldClim*.vrt files. Then try to download "WorldClim" again.

[Uriwolkow]

I cleaned the .vrt files successfully, and tried downloading again.
I keep getting the curl error on the pastclim 2.0.0 version.
And using 2.1.0, it started downloading, and again by 100% it produces the same fail error (even though I tried the 5m resolution this time)

download_dataset(dataset = "WorldClim_2.1_5m")
 [100%] Downloaded 179563774 bytes...
Error in eval(parse(text = file_details$download_function))(dataset = dataset,  : 
  something went wrong setting up this dataset
 the dataset will need downloading again

[dramanica]

Can you please post the content of the data dir?

[Uriwolkow]

these are the current contents, after the last download attempt:

> dir(get_data_path())
> list.files("C:/Users/University of Haifa/Documents/Uri_Albatross/Bubal_SDMs/models/data/pastclim")
 [1] "Beyer2020_annual_vars_v1.2.2.nc"   "example_climate_v1.3.0.nc"        
 [3] "worldclim_2.1"                     "WorldClim_2.1_5m_bio01_v1.4.0.vrt"
 [5] "WorldClim_2.1_5m_bio02_v1.4.0.vrt" "WorldClim_2.1_5m_bio03_v1.4.0.vrt"
 [7] "WorldClim_2.1_5m_bio04_v1.4.0.vrt" "WorldClim_2.1_5m_bio05_v1.4.0.vrt"
 [9] "WorldClim_2.1_5m_bio06_v1.4.0.vrt" "WorldClim_2.1_5m_bio07_v1.4.0.vrt"
[11] "WorldClim_2.1_5m_bio08_v1.4.0.vrt" "WorldClim_2.1_5m_bio09_v1.4.0.vrt"
[13] "WorldClim_2.1_5m_bio10_v1.4.0.vrt" "WorldClim_2.1_5m_bio11_v1.4.0.vrt"
[15] "WorldClim_2.1_5m_bio12_v1.4.0.vrt" "WorldClim_2.1_5m_bio13_v1.4.0.vrt"
[17] "WorldClim_2.1_5m_bio14_v1.4.0.vrt" "WorldClim_2.1_5m_bio15_v1.4.0.vrt"
[19] "WorldClim_2.1_5m_bio16_v1.4.0.vrt" "WorldClim_2.1_5m_bio17_v1.4.0.vrt"
[21] "WorldClim_2.1_5m_bio18_v1.4.0.vrt" "WorldClim_2.1_5m_bio19_v1.4.0.vrt"

[dramanica]

Ok. Just out of curiosity, what happens if you try to use the dataset.

[Uriwolkow]

"aware" of its existence or expecting it, but unavailable:

> land_mask <-
+   get_land_mask(time_ce = 1985, dataset = "WorldClim_2.1_5m") # Edited this from 1985 Worldclim to 1950 Beyer2020, 24.6.24
Error in check_var_downloaded(bio_variables, dataset) : 
  variable (bio01) not yet downloaded, use `download_dataset()`

[dramanica]

Ok, this is really strange, as we can see the bio01 file in your data path. Would you be willing to Zoom so that we can try a few things live? Send me an email on [email protected] and we can set up a session next week to get to the bottom of this strange behaviour.
Also, what do you get if you type:
Sys.getlocale()

[dramanica]

Note for future similar issues: This was due to a failed upgrade of the package which had left a messed up environment. A clean update eventually solved the problem.