ropensci / modistsp Goto Github PK
View Code? Open in Web Editor NEWAn "R" package for automatic download and preprocessing of MODIS Land Products Time Series
Home Page: https://docs.ropensci.org/MODIStsp
License: GNU General Public License v3.0
An "R" package for automatic download and preprocessing of MODIS Land Products Time Series
Home Page: https://docs.ropensci.org/MODIStsp
License: GNU General Public License v3.0
I am new to this package and having some issues opening up the interactive GUI.
When I type the command:
MODIStsp()
I get this error message:
GDAL version in use:2.0.1
Error in fromJSON(content, handler, default.size, depth, allowComments, :
invalid JSON input
I don't have any experience with JSON and can't seem to find a helpful post elsewhere.
Any help is much appreciated!
Good morning
Is possible to use the tool to process HDF data that have been a priori downloaded from EARTHDATA ?
Best
New user, managed with lots of help to install MODIStsp
on mac os x sierra, R 3.4.
Gui loads fine, but when I click on load options
and select already downloaded HDF files for MOD13C1 (eg.: MOD13C1.A2000209.006.2015147152824.hdf) from my external hard-drive, I get the following errors:
(R:3198): Gtk-WARNING **: Attempting to store changes into `/Users/magednosshi/.local/share/recently-used.xbel', but failed: Failed to create file “/Users/magednosshi/.local/share/recently-used.xbel.FQHL3Y”: No such file or directory
(R:3198): Gtk-WARNING **: Attempting to set the permissions of `/Users/magednosshi/.local/share/recently-used.xbel', but failed: No such file or directory
Is there a step I am missing in configuring the install?
Thanks,
~m
If I convert an hdf file directly in R to a selection of tiff files using gdal_translate, then look at the values in the 8th band (the sur_refl_qc band) I get different values in the .tiff file for when i do the same process with MODIStsp. Specifically, the last 2 values in the 32 bit string are different in the surf_refl_qc band.
If using MODIStsp this is a problem for me as when checking the quality of the bands in a tile, as a processing step i have previously looked for those pixels where the bit combination in the first 2 values of the 32 bit description is 11 or 10 - yet using MODIStsp the first 2 values are always 00, when in the gdal_translate converted file for the same pixels they do not have 00. Can you explain why these differences occur?
test if the use of the clusterR function allows to reduce processing time. If so, add it as an optional feature.
[ ] Explore chunked processing over large rasters
Hi
I experience this error when i run the installation code
Error: Command failed (1)
Steve
Use case: Building the tool into an automated pipeline. Specifying area of interest (AOI) by bbox parameter (long/lat) only AND bbox doesn't intersect with start_x/y end_x/y. We don't know what start_x/y end_x/y are in this use case.
GUI response: "The selected tiles does not intersect the output bounding box. Do you want to automatically retrieve the required tiles?"
My response: "Yes please :)"
Non-interactive mode: (1) Download tile specified by start_x/y end_x/y (2) Cut to bbox (3) if these do not intersect, then no data is returned in processed tif.
My response: oh crap!
Reason: the logic that the GUI uses in its dialogue isn't implemented as a keyword in the options_file.
Relevance: I think this use case is quite important as it is easy to specify AOI by lon/lat whereas very difficult to precompute which MODIS tile your AOI intersects. The logic already seems to be implemented just not accessible in non-interactive mode.
Thanks for a great product!
Hi,
Novice user here - I'm trying to install MODIStsp following the vignette instructions. I'm working on a MAC 10.12.5, and am running R 3.4.0 - I'm not sure that these will work because I am coming up against a number of missing dependency packages from which I cannot access via CRAN on R. If I go directly to Cran there are versions but not for the most recent Macs, I download the older ones and then try to install via install.packages ("XX", lib="~/XXX"), they appear to install. Similarly, I can't access MODIStsp via the cran directory on r - so I do the above action to install and it does say its installed, but when I try to then start the GUI - using library("MODIStsp") it says that it can't find the package. Where is my error and/or should I remove the version of r and try an older one?
Thanks
How to mosaic different MODIS titles datas using MODIStsp?
Hi
I am getting following error when I execute following code;
MODIStsp()
GDAL version in use:2.0.1
[Wed May 17 09:39:38 2017] Retrieving list of files from NASA server
Error in curl::curl_fetch_memory(url, handle = handle) :
Timeout was reached
Trying to reach ftp server - attempt 1
Error in curl::curl_fetch_memory(url, handle = handle) :
Timeout was reached
Trying to reach ftp server - attempt 2
Error in curl::curl_fetch_memory(url, handle = handle) :
Operation was aborted by an application callback
Trying to reach ftp server - attempt 3
Error in curl::curl_fetch_memory(url, handle = handle) :
Timeout was reached
Trying to reach ftp server - attempt 4
Error in curl::curl_fetch_memory(url, handle = handle) :
Timeout was reached
Information for my R session is:
R version 3.4.0 (2017-04-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows Server 2008 R2 x64 (build 7601) Service Pack 1
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] MODISTools_0.95.1 MODIStsp_1.3.2 gWidgetsRGtk2_0.0-83 cairoDevice_2.24 gWidgets_0.0-54
[6] RGtk2_2.20.33
loaded via a namespace (and not attached):
[1] hash_2.2.6 Rcpp_0.12.10 magrittr_1.5 raster_2.5-8 gdalUtils_2.0.1.7 lattice_0.20-35
[7] R6_2.2.1 foreach_1.4.3 stringr_1.2.0 plyr_1.8.4 httr_1.2.1 xts_0.9-7
[13] tools_3.4.0 rgdal_1.2-7 grid_3.4.0 data.table_1.10.4 pacman_0.4.6 R.oo_1.21.0
[19] rgeos_0.3-23 iterators_1.0.8 RJSONIO_1.3-0 R.utils_2.5.0 bitops_1.0-6 codetools_0.2-15
[25] curl_2.6 RCurl_1.95-4.8 sp_1.2-4 stringi_1.1.5 compiler_3.4.0 R.methodsS3_1.7.1
[31] XML_3.98-1.7 zoo_1.8-0
Bug found on MODIStsp_extract if shapefile partially outside of MODIS footprint --> implement changes made in lb_fastzonal
GUI loading sometimes "hangs" while searching gdal installation OR doesn't find the installation even if it's there.
Does not launch. Instead gives the following output on first launch:
> MODIStsp()
Loading required namespace: gWidgetsRGtk2
Loading required package: gWidgetsRGtk2
Loading required package: RGtk2
Loading required package: gWidgets
Loading required package: cairoDevice
Scanning for GDAL installations...
Checking Sys.which...
GDAL version 2.1.0
GDAL version in use: 2.1.0
Error in UseMethod("xmlAttrs", node) :
no applicable method for 'xmlAttrs' applied to an object of class "NULL"
>
And just the following message from then on:
> MODIStsp()
GDAL version in use: 2.1.0
Error in UseMethod("xmlAttrs", node) :
no applicable method for 'xmlAttrs' applied to an object of class "NULL"
>
I have installed the latest (as of October 25th, 2016) devel version in two separate computers (one with Xubuntu Linux 16.04 64-bit, the other one with Windows Vista 64-bit), both with R 3.3.1. Used to work well before...
install_github("lbusett/MODIStsp")
Downloading GitHub repo lbusett/MODIStsp@master from URL https://api.github.com/repos/lbusett/MODIStsp/zipball/master
Installing MODIStsp
"D:/R-321.3/R-331.3/bin/x64/R" --no-site-file --no-environ
--no-save --no-restore --quiet CMD INSTALL
"C:/Users///***/Temp/RtmpAZ4RWF/devtools119c2f747d1d/lbusett-MODIStsp-3e7e04b"
--library="D:/R-3.2.3/R-3.3.3/library" --install-tests
At this point I receive a pop-up window informing me that the program can't start because libatk-1.0-0.dll is missing. The pop-up window gets stuck and I need to kill the R session.
However: when this pop-up appered the first time, I did install GTK+. I did make sure the PATH variables (both system and user) were updated. I restarted both R and the system. I tried installing GTK+ manually to bypass the issue, but no improvement.
Any advice?
Thank you,
Effie
[1] "D:\SierraNevadaMODIS_LST\AOI.shp"
[Thu Jun 29 18:19:48 2017] Retrieving the Extent, please wait... Error in if (digits < 0L) digits <- 6L else { : missing value where TRUE/FALSE needed Attached the shapefile (created in ArcGIS).
Any idea ?
SierraNevadaMODIS_LST.zip
Simplify the structure of RData files in order not to include information of MODIS products, but only parameters.
Introduce check in correct size of ftp/http download
Why does this error come out?
It is the internet?
library(devtools)
install_github("lbusett/MODIStsp")
Error in curl::curl_fetch_disk(url, x$path, handle = handle) :
Timeout was reached
To install RGtk2 for MODIStsp
using R version 3.3.3 on Platform: x86_64-w64-mingw32/x64 (64-bit)
Executing
library(gWidgetsRGtk2)
This error comes out:
FAILED TO LOAD RGtk2 dynamic library
How do I solve this?
Thank you
While changing from 'Sinusoidal' to 'User defined', gives the following error, which prevents the user from inserting the proj4 string for the user defined projection:
Error in paste(...) : argument "message" is missing, with no default
Changing the proj4 string and bbox directly from the JSON file, and then loading it on the GUI, allows for the processing to run, but the message appears once more.
Modify MOD13Q!, etcetera to M*13Q1 etcetera
Hi, tried to install the development version on a mac from github using
library(devtools)
install_github("lbusett/MODIStsp", ref = "master")
and received the following
Error: Could not find build tools necessary to build gWidgets
Is there an additional step I am missing?
Thanks
Allow the use of download accelerator "aria2c" to reduce download time
Organize the time series in subfolder by layer and platform
Hello all,
came across this package, quite nice one. Have installed it and is working now.
My question is, is there any function that allows for pausing and and resuming the download and processing?
I have looked through decription in the package and alos in the closed issues in this forum but did not find any clue.
I will appreciate if someone can point me in the right direction.
Thanks for you assistance.
I tried several times to test MODISstp. But i was not success. Already,
i find when i select reprojection section, choose LatLonWGS84,
i wrote wrong value for pixel size. I will be appreciate if send me the link that i find this values
At the moment, specifying start date 2005-03-01 and end date 2015-06-30 will process ALL images between the two dates, but sometimes it can be useful to download only one "season", but for multiple years.
Need to make so that the user can do "splitted" download, so that, for example, data BETWEEN 03-01 and 06-03 for MULTIPLE YEARS can be downloaded without downloading anything else
Snow cover products are not present in the usual "suite" of Land products, since they are distributed by NSIDC (https://nsidc.org/data/mod10a1#).
Products MD10A1 (Daily), MD10A2 (8-Day) and MOD10C2 (8-day 0.05deg) should be added to the "pool".
I got this error message
Error: package ‘RGtk2’ required by ‘gWidgetsRGtk2’ could not be found
In addition: Warning message:
package ‘gWidgetsRGtk2’ was built under R version 3.3.3
i tried all method you suggested including
http://ftp.gnome.org/pub/gnome/binaries/win64/gtk+/2.22/gtk+-bundle_2.22.1-20101229_win64.zip (OR
http://ftp.gnome.org/pub/gnome/binaries/win32/gtk+/2.22/gtk+-bundle_2.22.1-20101227_win32.zip if on
Win32)
,
Can you explain this further maybe i made some mistake while specifying environmental variable...
unzip the archive on a folder of your choice (e.g., C:\Program Files\GTK+), then add the path to the
“bin” sub-folder (e.g., C:\Program Files\GTK+\bin\ to your system PATH environment variable.
Restart your system and try loading again gWidgetsRGtk2
trying to process some MCD12Q1 LandCover data, but got wrong result.
R x64 3.3.2
MODIStsp v1.3.0
Got nothing output result but a hdr file "MCD12Q1_LC1_2013_001.hdr" and content is "data ignore value = 255 "
When i use GUI to download image in download only first image form that time period, however when i use command in R script without GUI it's can load multiple image.
Can someone help me FIX this.
For non-tiled products:
--> Selection of tiles should be disabled
--> Native projection should be set to Lat/Lon WGS84
--> Native resolution should be set ot 0.05 (Modify XML file or convert from 5600 to 0.05)
R x64 3.3.1, Win7 x64, stable version of MODIStsp
other modis product work well,but for MOD13C2, each exported data folder (for exsample: VI_Monthly_005dg\b2_NIR) only hdr file was there(MOD13C2_b2_NIR_2000_032.hdr), no data file.
Here are the error message:
Scanning for GDAL installations...
Checking Sys.which...
Checking common locations...
GDAL version 2.1.2
GDAL version in use: 2.0.1
[1] "E:\TS\MOD13C2\TS"
[1] "E:\TS\MOD13C2\HDF"
[1] "E:\TS\MOD13C2\World_2000-2008"
No encoding supplied: defaulting to UTF-8.
[Fri Oct 28 11:39:52 2016]Downloading Terra Files for date 2000_02_01 : 1 of 1
No encoding supplied: defaulting to UTF-8.
[Fri Oct 28 11:41:43 2016]1 files for date of 2000.02.01 were successfully downloaded!
[Fri Oct 28 11:41:43 2016] Processing Terra VI_QA files for date: 2000_02_01
[Fri Oct 28 11:41:48 2016] Processing Terra b1_Red files for date: 2000_02_01
[Fri Oct 28 11:41:52 2016] Processing Terra b2_NIR files for date: 2000_02_01
[Fri Oct 28 11:41:57 2016] Processing Terra Rely files for date: 2000_02_01
[Fri Oct 28 11:42:01 2016]Computing Terra GV for date: 2000_02_01
Error in .local(.Object, ...) :
`E:\TS\MOD13C2\TS\VI_Monthly_005dg\b1_Red\MOD13C2_b1_Red_2000_032.dat' does not exist in the file system,
and is not recognised as a supported dataset name.
In addition: Warning messages:
1: In memory.limit(8000) : cannot decrease memory limit: ignored
2: running command '"C:\OSGeo4W64\bin\gdalbuildvrt.exe" -sd "3" -srcnodata "65535" -vrtnodata "65535" "C:\Users\SHI138\AppData\Local\Temp\RtmpCaOGIO\file2e9c6a8240e.vrt" "E:\TS\MOD13C2\HDF/MOD13C2.A2000032.005.2006272104028.hdf"' had status 1
3: running command '"C:\OSGeo4W64\bin\gdal_translate.exe" -a_nodata "65535" -ot "UInt16" -of "ENVI" -a_srs "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs" "C:\Users\SHI138\AppData\Local\Temp\RtmpCaOGIO\file2e9c6a8240e.vrt" "E:\TS\MOD13C2\TS/VI_Monthly_005dg/VI_QA/MOD13C2_VI_QA_2000_032.dat"' had status 1
4: running command '"C:\OSGeo4W64\bin\gdalbuildvrt.exe" -sd "4" -srcnodata "-1000" -vrtnodata "32767" "C:\Users\SHI138\AppData\Local\Temp\RtmpCaOGIO\file2e9c173b48a6.vrt" "E:\TS\MOD13C2\HDF/MOD13C2.A2000032.005.2006272104028.hdf"' had status 1
5: running command '"C:\OSGeo4W64\bin\gdal_translate.exe" -a_nodata "32767" -ot "Int16" -of "ENVI" -a_srs "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs" "C:\Users\SHI138\AppData\Local\Temp\RtmpCaOGIO\file2e9c173b48a6.vrt" "E:\TS\MOD13C2\TS/VI_Monthly_005dg/b1_Red/MOD13C2_b1_Red_2000_032.dat"' had status 1
6: running command '"C:\OSGeo4W64\bin\gdalbuildvrt.exe" -sd "5" -srcnodata "-1000" -vrtnodata "32767" "C:\Users\SHI138\AppData\Local\Temp\RtmpCaOGIO\file2e9c11783805.vrt" "E:\TS\MOD13C2\HDF/MOD13C2.A2000032.005.2006272104028.hdf"' had status 1
7: running command '"C:\OSGeo4W64\bin\gdal_translate.exe" -a_nodata "32767" -ot "Int16" -of "ENVI" -a_srs "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs" "C:\Users\SHI138\AppData\Local\Temp\RtmpCaOGIO\file2e9c11783805.vrt" "E:\TS\MOD13C2\TS/VI_Monthly_005dg/b2_NIR/MOD13C2_b2_NIR_2000_032.dat"' had status 1
8: running command '"C:\OSGeo4W64\bin\gdalbuildvrt.exe" -sd "13" -srcnodata "-1" -vrtnodata "255" "C:\Users\SHI138\AppData\Local\Temp\RtmpCaOGIO\file2e9ce0c6d55.vrt" "E:\TS\MOD13C2\HDF/MOD13C2.A2000032.005.2006272104028.hdf"' had status 1
9: running command '"C:\OSGeo4W64\bin\gdal_translate.exe" -a_nodata "255" -ot "Byte" -of "ENVI" -a_srs "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs" "C:\Users\SHI138\AppData\Local\Temp\RtmpCaOGIO\file2e9ce0c6d55.vrt" "E:\TS\MOD13C2\TS/VI_Monthly_005dg/Rely/MOD13C2_Rely_2000_032.dat"' had status 1
Error in .rasterObjectFromFile(x, band = band, objecttype = "RasterLayer", :
Cannot create a RasterLayer object from this file. (file does not exist)
MODIStsp is very helpful tools to download and pre-process MODIS images, moreover, it also includes vegetation indices calculation. However, VI calculation from MCD43A4 (BRDF reflectance data) is still missing from the options. I hope the next version will facilitate this calculation. I thank you in advance for the effort of the developer(s).
From HG issue reported here, #64
With the hotfix version download speed was reduced dramatically and I got the error
Warning #messages:
1: running command '"aria2c"' had status 1I have just downloaded the "devel" version and I am still getting the same warning message
1: running command '"aria2c"' had status 1
I'm trying to download MODIS data using the spatial_file_path
-option. When using the shapefile manually in the GUI to "Load extent from a spatial file", everything works as expected. However, when I provide it as an argument like this
MODIStsp(gui = FALSE, options_file = "options.json", spatial_file_path = "shapefile.shp")
I get the following error:
Error in as.vector(x) : no method for coercing this S4 class to a vector
Would you have any ideas what might be the problem here?
With HTTP
Trying to reach http server - attempt 25
Error in curl::curl_fetch_memory(url, handle = handle) :
.
.
.
How much time do I need to download a scene?
Connection #0 to host ladsweb.nascom.nasa.gov left intact
[Tue Apr 04 16:51:54 2017] Downloading Terra Files for date 2000_02_18 : 1 of 1
|================= | 18%
Error in curl::curl_fetch_memory(url, handle = handle) :
Timeout was reached
In addition: There were 37 warnings (use warnings() to see them)
[Tue Apr 04 17:11:54 2017] Download Error - Retrying...
[Tue Apr 04 17:11:55 2017] Downloading Terra Files for date 2000_02_18 : 1 of 1
Error in curl::curl_fetch_memory(url, handle = handle) :
Timeout was reached
[Tue Apr 04 17:12:08 2017] Download Error - Retrying...
[Tue Apr 04 17:12:09 2017] Downloading Terra Files for date 2000_02_18 : 1 of 1
|============= | 15%
Until now I can not download any image
Allow computing the VIs and SIs by chunks of lines to avoid memory problems if large mosaics (i.e., multiple tiles) are required.
I ran into an error for the products MCD12Q1 and MCD64A1. Downloading and processing seems to work fine, until this step:
Creating Virtual Files and rts time series for layer LC1
This error occurs:
Error in grep(file_prefix, out_meta_files) : object 'file_prefix' not found
Interestingly, the processing step and the following error also occur when both Create Virtual Rasters
and Create RasterStacks
are set to No
.
I'm using version 1.3.2.9 by the way.
using R version 3.2.2 on Platform: x86_64-w64-mingw32/x64 (64-bit)
The tiff files for 8-day LAI/FPAR (LAI_1km and FPAR_1km) are not created on the output folders (i.e., folders FAPR and LAI are empty). It does write Tiff files for QC bits.
e.g., change "=" to <-
single/double quotes
etc.
from @maged:
I have a question about how you create the and save the Raster stack as
RData..
When I create the stack:
s <- stack(dir())
save(s,file="~/Documents/PUBS/EVI RDATA/s.Rdata")
then look at the attributes
s
I get everything as I would with the RData file created by MODIStsp, but
not the "time", for example:
time : 2000-07-27 - 2017-06-26 (range)
How can I add this when creating the Rasterstack?
Thanks
maged
With reference to this:
Also, if I may make a suggestion, you could add some notes on the syntax to use on a custom index, e.g. the correct names of the bands to be used in order for it to work correctly (in my opinion, simply "b1, ..., b7" would do, but "Red, NIR, ..., SWIR2" would work fine as well). in #51
, improve instructions on additional indexes specifications
I'm unable to download M*D17A3 product (layer: Gridded 1 Km Annual Net Productivity). I've tried ftp option:
[Thu Feb 23 14:41:01 2017] Retrieving list of files from NASA server
Trying to reach ftp server - attempt 1
...and http option:
[Thu Feb 23 14:41:38 2017] Retrieving list of files from NASA server
Trying to reach http server - attempt 1
While both of messages claim that server is down, however I can access https://e4ftl01.cr.usgs.gov/ without any problems. I also was able to download a different product - the problem occurs only with M*D17A3.
Thanks for your work on this package and let me know if you need some more information.
GDAL version 2.1.1
Error in MODIStsp() :
Your local GDAL installation does not support HDF4 format. Please install HDF4 support and recompile GDAL.
I use R in windows 7, and this package till yesterday had worked. But today it get this error.
i download HDF4 and install in windows, but i don`t know what can i do for the next step.
I will gratify if anybody help me.
I have been unable to connect to the server, for both ftp and http connection options. It shows me this error:
Trying to reach http server - attempt 1
Error in curl::curl_fetch_memory(url, handle = handle) :
Couldn't connect to server
This error continues to come up 50-100x.
I've tried the 3-4 fixes listed under your troubleshooting section, none of which have worked thus far.
My internet is really really slow, ie 0.6 mbps, could this be the issue?
I'm also running internet off of a cellular hotspot, could there be special proxy settings associated with this?
Thanks for the help!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.