lifewatch/lwdataexplorer

R package to access the data from LifeWatch Data Explorer

lwdataexplorer's Issues

Improve feedback when authentication fails

Currently there's nothing telling the user that the authentication failed besides the warning about a different query applied in the server compared to the client.

categories 'all' not consistent

Raised by Steven Pint

Comments LW package:
getStationData() -> categories = "All" ==> doesn't work (in info example it is like this)
getStationData() -> categories = "all" ==> works but no pigments data

Also selecting station 'W09' provides station 'W09' and 'w09' -> I assume it is the same station but can be confusing. Although, it's easily solved in R so no big issue, just mentioning it.

Issue retrieving CPOD data with GetCpodData

Documentation states:

quality One of ("Hi","Mod", "Lo", "Hi+Mod") .....

Data is available when processing is set to "Validated":

df_validated <- getCpodData("2020-04-19", "2020-04-21", processing = "Validated", quality = "Hi+Mod", by = "1 week")

Quality "Hi+Mod" does not retrieve data when processing is set to 'Raw':

df_raw <- getCpodData("2020-04-19", "2020-04-21", processing = "Raw", quality = "Hi+Mod", by = "1 week")

Resulting in:

[...]
Warning message:
In outputQC(input, out) : No data returned. Try relaxing query parameters.

Validated is by default "Hi+Mod". That class does not exist for "Raw", and each quality needs to be specified in a vector (quality = c("Hi", "Mod")).

Add pagination

ctd-data function missing?

There is a ctd data explorer, but no function for retrieving the data in the lwdataexplorer R package? https://rshiny.vsc.lifewatch.be/ctd-data/

help file getStationData

In the arguments section of the help file of getStationData "..." refers to parameters passed to lw_check_lwdataserver().

But lw_check_lwdataserver is not an exported function in the package and has no help file. As a consequence, an end-user cannot know which other parameters can be sent to lw_check_dataserver.

database connection does not work on rstudio.lifewatch.be

Hi all, I wanted to report two different error messages while trying to retrieve data on rstudio.lifewatch.be

 > zooscandata <- getZooscanData("2011-01-01", "2021-04-14") # Only data
- Query mode: Database connection
Error in (function (uri = "mongodb://127.0.0.1", pem_file = NULL, pem_pwd = NULL,  : 
  failed to parse URI: 0 (Invalid URI Schema, expecting 'mongodb://' or 'mongodb+srv://')
Error in (function (uri = "mongodb://127.0.0.1", pem_file = NULL, pem_pwd = NULL,  : 
  failed to parse URI: 0 (Invalid URI Schema, expecting 'mongodb://' or 'mongodb+srv://')
Error in lw_mdf_is_string(mdf, input, par) : No connection to mongodb
> stationdata <- getStationData("2019-07-22", "2019-07-23", stations = "all", categories = "all")
- Query mode: Database connection
Date passed as text. Trying to transform to date
ERROR: The server seems unavailable. Please contact the package maintainer if the issue persists: <[email protected]>
Warning message:
In lw_mdf_is_string(mdf, input, par) :
  Something unexpected happened. Check the server logs.

My environment:

> sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.6 LTS

Matrix products: default
BLAS:   /usr/lib/libblas/libblas.so.3.0
LAPACK: /usr/lib/lapack/liblapack.so.3.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] lwdataexplorer_0.0.0.9000 lifecycle_1.0.1          

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.7         magrittr_2.0.1     tidyselect_1.1.0   xtable_1.8-4       R6_2.5.1           rlang_0.4.12      
 [7] fastmap_1.1.0      fansi_0.5.0        stringr_1.4.0      dplyr_1.0.2        tools_3.6.3        packrat_0.5.0     
[13] utf8_1.2.2         cli_3.1.0          htmltools_0.5.2    ellipsis_0.3.2     digest_0.6.29      tibble_3.1.6      
[19] mongolite_2.4.1    crayon_1.4.2       shiny_1.7.1        later_1.3.0        purrr_0.3.4        lwdataserver_0.1.0
[25] tidyr_1.1.2        promises_1.2.0.1   RODBC_1.3-16       vctrs_0.3.8        bitops_1.0-7       RCurl_1.98-1.5    
[31] mime_0.12          glue_1.5.1         stringi_1.7.6      compiler_3.6.3     pillar_1.6.4       generics_0.0.2    
[37] jsonlite_1.7.2     httpuv_1.6.3       pkgconfig_2.0.3   

access restriction - 403 Forbidden - You don't have permission to access this resource

Hi all
There seems to be an access restriction error for accessing the data:
These examples

library(lwdataexplorer)
getCpodData("2020-04-19", "2020-04-21", processing = "Validated", by = "1 week")
getStationData("2019-07-22", "2019-07-23", stations = "all", categories = "all")
getBatsData("2014-08-01", "2014-09-01", "1 min")

give all the same error (on local windows pc, rstudio.lifewatch.be and rstudio.vsc.lifewatch.be):

- Query mode: Post request to OpenCPU server
Error : lexical error: invalid char in json text.
                                       <!DOCTYPE HTML PUBLIC "-//IETF/
                     (right here) ------^

In addition: Warning message:
In lw_check_config(...) :
  Configuration details are missing: MONGODB_CONN, MONGODBSANT1_CONN, ODBC_USER, ODBC_PSWD, ETN_ODBC_DSN, ETN_SERVER, ETN_DBNAME, UVA_ODBC_DSN, UVA_SERVER, UVA_DBNAME, EUROBIS_ODBC_DSN, EUROBIS_SERVER, EUROBIS_DBNAME. Query redirected to OpenCPU.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
Error in lw_output_qc(input, out) : 
  is.list(out) | is.character(out) is not TRUE

Improve error messages and documentation about direct database querying

Triggered by #10

Using lwdataexplorer from rstudio.lifewatch.be requires of extra configuration, and having access to some secrets.

• Improve error messages when secrets are not available a9f8131
• Add logic to query through OpenCPU instead a9f8131
• Improve documentation - README and a new vignette

Error in curl fetch memory

Dear developer,

I am trying to use the get function to access the data. I get this error

getBatsData("2014-08-01", "2014-09-01", "1 min")
# Error in curl::curl_fetch_memory(url, handle = handle) : gnutls_handshake() failed: Handshake failed

sessionInfo()
# R version 3.4.4 (2018-03-15)
# Platform: x86_64-pc-linux-gnu (64-bit)
# Running under: Ubuntu 14.04.5 LTS
# 
# Matrix products: default
# BLAS: /usr/lib/libblas/libblas.so.3.0
# LAPACK: /usr/lib/lapack/liblapack.so.3.0
# 
# locale:
#   [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
# [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
# [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
# [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
# [9] LC_ADDRESS=C               LC_TELEPHONE=C            
# [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
# 
# attached base packages:
#   [1] datasets  stats     graphics  grDevices utils     methods   base     
# 
# other attached packages:
#   [1] RPostgreSQL_0.6-2 DBI_1.1.1         XML_3.99-0.3     
# [4] httr_1.4.2        googleVis_0.6.6  
# 
# loaded via a namespace (and not attached):
#   [1] Rcpp_1.0.6        compiler_3.4.4    highr_0.8        
# [4] prettyunits_1.1.1 remotes_2.2.0     tools_3.4.4      
# [7] testthat_2.3.2    digest_0.6.26     pkgbuild_1.1.0   
# [10] pkgload_1.1.0     evaluate_0.14     jsonlite_1.7.2   
# [13] memoise_1.1.0     rlang_0.4.11      reprex_2.0.1     
# [16] cli_2.5.0         rstudioapi_0.11   yaml_2.2.1       
# [19] xfun_0.18         withr_2.3.0       knitr_1.30       
# [22] desc_1.2.0        fs_1.3.1          devtools_2.3.2   
# [25] rprojroot_1.3-2   glue_1.4.2        R6_2.5.0         
# [28] processx_3.5.2    rmarkdown_2.5     sessioninfo_1.1.1
# [31] callr_3.7.0       magrittr_2.0.1    htmltools_0.5.0  
# [34] backports_1.1.10  codetools_0.2-15  ps_1.4.0         
# [37] ellipsis_0.3.2    usethis_1.6.3     assertthat_0.2.1 
# [40] crayon_1.4.1     

getEtnData() - invalidresponse from an upstream server

Received by email

Hi Salva,

(....) I prefer to retrieve the data in R directly using the lwdataexplorer package. Unfortunately, this gives me an error message still.

Could you check whether this call to retrieve data from the ETN is correct?:

getEtnData(startdate = "2020-10-01 12:00:00", stopdate = "2022-01-31 12:00:00", action = "Detections", by = "1 min", networks = "All", projects = "All", usr = "xxx", pwd = "xxx", params = FALSE)

It gives me the following error message:

- Query mode: Post request to OpenCPU server
Error : lexical error: invalid char in json text.
                                             <!DOCTYPE HTML PUBLIC "-//IETF/
                     (right here) ------^<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>
Fehler in lw_output_qc(input, out) : 
  is.list(out) | is.character(out) is not TRUE

Any idea what I did wrong here?

Best and many thanks for your help!