expersso/bis

Programmatic access to BIS data

bis's Issues

new data structure at BIS?

There seems to be a mess-up with the new data structure at BIS.
They note that

24 February 2022
CSV files have been renamed according to the new Dataset Identifiers, find the new list here. For previously stored URLs, updating the path will be necessary.
(ie for EER, old path https://www.bis.org/statistics/full_webstats_eer_d_dataflow_csv_col.zip has been changed to new path https://www.bis.org/statistics/full_eer_d_csv_col.zip).

see their homepage

The code (package version 0.2.1) produces now an empty df:

> require(BIS)
> datasets <- get_datasets()
> cons_data <- get_bis(datasets$url[datasets$name == "Consolidated banking statistics"] , quiet = TRUE)

Warning messages:
1: `as_data_frame()` was deprecated in tibble 2.0.0.
Please use `as_tibble()` instead.
The signature and semantics have changed, see `?as_tibble`.
This warning is displayed once every 8 hours.
Call `lifecycle::last_lifecycle_warnings()` to see where this warning was generated. 
2: The `x` argument of `as_tibble.matrix()` must have unique column names if `.name_repair` is omitted as of tibble 2.0.0.
Using compatibility `.name_repair`.
This warning is displayed once every 8 hours.
Call `lifecycle::last_lifecycle_warnings()` to see where this warning was generated. 

> head(cons_data)
# A tibble: 0 x 14
# ... with 14 variables: frequency <chr>, measure <chr>, reporting_country <chr>, cbs_bank_type <chr>, cbs_reporting_basis <chr>, balance_sheet_position <chr>,
#   type_of_instruments <chr>, remaining_maturity <chr>, currency_type_of_booking_location <chr>, counterparty_sector <chr>, counterparty_country <chr>,
#   collection_indicator <lgl>, date <chr>, obs_value <dbl>

get_bis() drops information during conversion

Summary:
When downloading the "Triennial Survey statistics on turnover" and "OTC derivatives outstanding" using get_bis(), information is lost during conversion of the original datasets. For example, the columns on "denomination leg 1" and "denomination leg 2" appear to be dropped, leading to incomplete datasets being imported.

Steps to reproduce:

library("BIS")
ds  <- get_datasets()

der <- get_bis(ds$url[grep("Triennial Survey statistics on", ds$name)])
otc <- get_bis(ds$url[grep("OTC derivatives outstanding", ds$name)])

Expected result:
The calls to get_bis() should run through and return the full datasets, with all columns intact.

What happens instead:
The data is downloaded and assigned to the objects before the "<-" operator. However, during the internal conversion process, information appears to become lost. To see this, compare the output of the colnames() commands below.

download.file(ds$url[grep("Triennial Survey statistics on", ds$name)],
              mode = "wb", destfile = "full_BIS_DER_csv.zip")
unzip("full_BIS_DER_csv.zip")
der2 <- read.csv("full_WEBSTATS_DER_DATAFLOW_csv.csv", header = TRUE,
                 stringsAsFactors = FALSE, skip = 10)
colnames(der)
colnames(der2)

download.file(ds$url[grep("OTC derivatives outstanding", ds$name)],
              mode = "wb", destfile = "full_bis_otc_csv.zip")
unzip("full_bis_otc_csv.zip")
otc2 <- read.csv("WEBSTATS_OTC_DATAFLOW_csv_col.csv", header = TRUE,
                stringsAsFactors = FALSE)
colnames(otc)
colnames(otc2)

Note that the function get_bis() also returns the below error message before completing:

Warning message:
In evalq(as.numeric(obs_value), <environment>) : NAs introduced by coercion

This message also appears when downloading the datasets "Global liquidity indicators", "Credit to the non-financial sector" and "Property prices: selected series".

System info:

sessionInfo()

R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252   
[3] LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C                           
[5] LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] BIS_0.2.1

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.0       rstudioapi_0.8   bindr_0.1.1      xml2_1.2.0       magrittr_1.5    
 [6] hms_0.4.2        tidyselect_0.2.5 rvest_0.3.2      R6_2.3.0         rlang_0.3.0.1   
[11] httr_1.3.1       dplyr_0.7.8      tools_3.5.1      yaml_2.2.0       assertthat_0.2.0
[16] tibble_1.4.2     crayon_1.3.4     bindrcpp_0.2.2   tidyr_0.8.2      purrr_0.2.5     
[21] readr_1.1.1      curl_3.2         glue_1.3.0       compiler_3.5.1   pillar_1.3.0    
[26] pkgconfig_2.0.2 

Thanks for the great package!

get_datasets() proxy issue

hello,
Like the same issue here (expersso/OECD#11) can you modify the package to support corporate proxy using httr:
the solution is to modify the get_datasets() function like below:

get_datasets <- function() {
url <- complete_url("/statistics/full_data_sets.htm")
page <- xml2::read_html(httr::GET(url))
nodes <- rvest::html_nodes(page, xpath = "//a[contains(@href, 'zip')]")
dplyr::tibble(name = rvest::html_text(nodes),
url = complete_url(rvest::html_attr(nodes, "href")))
}

Dataset not working

Hey, i'm an econ student and I've been using the package, but recently it seems one of the datasets stopped working. I can get others but the one i'm interested in can't seem to load, i don't think it's a local problem. It stopped working a few weeks ago. It's: 5 Credit to the non-financial sector - https://www.bis.org/statistics/full_bis_total_credit_csv.zip

get_datasets drops amount outstanding measure in locational banking statistics

> bis_location <- get_bis(bis_datasets$url[bis_datasets$name == "Locational banking statistics"], quiet = TRUE)
Parsed with column specification:
cols(
  .default = col_character()
)
See spec(...) for full column specifications.
|====================================================================================================| 100%  337 MB
Parsed with column specification:
cols(
  .default = col_double(),
  FREQ = col_character(),
  Frequency = col_character(),
  L_MEASURE = col_logical(),
  Measure = col_character(),
  L_POSITION = col_character(),
  `Balance sheet position` = col_character(),
  L_INSTR = col_character(),
  `Type of instruments` = col_character(),
  L_DENOM = col_character(),
  `Currency denomination` = col_character(),
  L_CURR_TYPE = col_character(),
  `Currency type of reporting country` = col_character(),
  L_PARENT_CTY = col_character(),
  `Parent country` = col_character(),
  L_REP_BANK_TYPE = col_character(),
  `Type of reporting institutions` = col_character(),
  L_REP_CTY = col_character(),
  `Reporting country` = col_character(),
  L_CP_SECTOR = col_character(),
  `Counterparty sector` = col_character()
  # ... with 6 more columns
)
See spec(...) for full column specifications.
|====================================================================================================| 100%  337 MB
Warning: 154340 parsing failures.
   row       col           expected   actual                                                                                                   file
141051 L_MEASURE 1/0/T/F/TRUE/FALSE S        '/var/folders/9p/q05l96xj0cl_z2vb96crjzqh0000gn/T//RtmpVvNWQC/WEBSTATS_LBS_D_PUB_DATAFLOW_csv_col.csv'
141051 1977-Q4   1/0/T/F/TRUE/FALSE 24.000   '/var/folders/9p/q05l96xj0cl_z2vb96crjzqh0000gn/T//RtmpVvNWQC/WEBSTATS_LBS_D_PUB_DATAFLOW_csv_col.csv'
141052 L_MEASURE 1/0/T/F/TRUE/FALSE S        '/var/folders/9p/q05l96xj0cl_z2vb96crjzqh0000gn/T//RtmpVvNWQC/WEBSTATS_LBS_D_PUB_DATAFLOW_csv_col.csv'
141052 1977-Q4   1/0/T/F/TRUE/FALSE 1060.000 '/var/folders/9p/q05l96xj0cl_z2vb96crjzqh0000gn/T//RtmpVvNWQC/WEBSTATS_LBS_D_PUB_DATAFLOW_csv_col.csv'
141053 L_MEASURE 1/0/T/F/TRUE/FALSE S        '/var/folders/9p/q05l96xj0cl_z2vb96crjzqh0000gn/T//RtmpVvNWQC/WEBSTATS_LBS_D_PUB_DATAFLOW_csv_col.csv'
...... ......... .................. ........ .......................................... [... truncated]

> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] grid      stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] BIS_0.2.1           fredr_1.1.0.9000    censusapi_0.7.1     htmlTable_2.1.0     compareDF_2.3.0    
 [6] blsAPI_0.2.1        pdftools_2.3.1      magrittr_1.5        timetk_2.4.0        seasonal_1.7.1     
[11] HDInterval_0.2.2    smoother_1.1        TTR_0.24.2          cansim_0.3.5        jsonlite_1.7.1     
[16] geofacet_0.2.0      zoo_1.8-8           janitor_2.0.1       ggrepel_0.8.2       openxlsx_4.2.2     
[21] readxl_1.3.1        openintro_2.0.0     usdata_0.1.0        cherryblossom_0.1.0 airports_0.1.0     
[26] httr_1.4.2          rvest_0.3.6         xml2_1.3.2          RCurl_1.98-1.2      pander_0.6.3       
[31] DT_0.16             scales_1.1.1        lubridate_1.7.9     forcats_0.5.0       stringr_1.4.0      
[36] dplyr_1.0.2         purrr_0.3.4         readr_1.4.0         tidyr_1.1.2         tibble_3.0.4       
[41] ggplot2_3.3.2       tidyverse_1.3.0    

loaded via a namespace (and not attached):
  [1] backports_1.1.10    selectr_0.4-2       lazyeval_0.2.2      sp_1.4-4            splines_4.0.2      
  [6] listenv_0.8.0       digest_0.6.26       htmltools_0.5.0     fansi_0.4.1         checkmate_2.0.0    
 [11] recipes_0.1.14      globals_0.13.1      modelr_0.1.8        gower_0.2.2         xts_0.12.1         
 [16] sandwich_3.0-0      askpass_1.1         rsample_0.0.8       jpeg_0.1-8.1        strucchange_1.5-2  
 [21] colorspace_1.4-1    blob_1.2.1          haven_2.3.1         xfun_0.18           crayon_1.3.4       
 [26] survival_3.2-7      glue_1.4.2          gtable_0.3.0        ipred_0.9-9         imguR_1.0.3        
 [31] qpdf_1.1            DBI_1.1.0           Rcpp_1.0.5          units_0.6-7         lava_1.6.8         
 [36] prodlim_2019.11.13  htmlwidgets_1.5.2   geogrid_0.1.1       RColorBrewer_1.1-2  x13binary_1.1.39-2 
 [41] ellipsis_0.3.1      pkgconfig_2.0.3     farver_2.0.3        nnet_7.3-14         dbplyr_1.4.4       
 [46] tidyselect_1.1.0    labeling_0.4.2      rlang_0.4.8         munsell_0.5.0       cellranger_1.1.0   
 [51] tools_4.0.2         cli_2.1.0           generics_0.0.2      broom_0.7.2         evaluate_0.14      
 [56] yaml_2.2.1          knitr_1.30          fs_1.5.0            zip_2.1.1           future_1.19.1      
 [61] nlme_3.1-149        compiler_4.0.2      rstudioapi_0.11     curl_4.3            png_0.1-7          
 [66] e1071_1.7-4         reprex_0.3.0        stringi_1.5.3       rgeos_0.5-5         lattice_0.20-41    
 [71] Matrix_1.2-18       classInt_0.4-3      urca_1.3-0          vctrs_0.3.4         pillar_1.4.6       
 [76] lifecycle_0.2.0     vars_1.5-3          furrr_0.2.0         lmtest_0.9-38       data.table_1.13.0  
 [81] bitops_1.0-6        R6_2.4.1            KernSmooth_2.23-17  gridExtra_2.3       codetools_0.2-16   
 [86] MASS_7.3-53         assertthat_0.2.1    rjson_0.2.20        withr_2.3.0         rnaturalearth_0.1.0
 [91] mgcv_1.8-33         parallel_4.0.2      hms_0.5.3           rpart_4.1-15        timeDate_3043.102  
 [96] class_7.3-17        rmarkdown_2.4       snakecase_0.11.0    sf_0.9-6            tinytex_0.26       

get_bis() gets stuck trying to download LBS, CBS, SEC data sets

Summary:
get_bis() gets stuck trying to download the "Locational banking statistics" (LBS), "Consolidated banking statistics" (CBS) and the "Debt securities statistics" (SEC) from the BIS's homepage.

Steps to reproduce:

library("BIS")
ds  <- get_datasets()

lbs <- get_bis(ds$url[grep("Locational banking statistics", ds$name)])
cbs <- get_bis(ds$url[grep("Consolidated banking statistics", ds$name)])
sec <- get_bis(ds$url[grep("Debt securities statistics", ds$name)])

Expected result:
The above calls to get_bis() should run through and assign the downloaded data to the objects before the "<-" operator.

What happened instead:
The download starts, showing a progress bar which gradually goes to 100%. After that, nothing happens for an extended period of time. After several minutes of inactivity, the system starts using up larger amounts of CPU resources and RAM, at which point I typically force-quit R/R-Studio.

System information:

sessionInfo()

R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252   
[3] LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C                           
[5] LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] BIS_0.2.1

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.0       rstudioapi_0.8   bindr_0.1.1      xml2_1.2.0       magrittr_1.5    
 [6] hms_0.4.2        tidyselect_0.2.5 rvest_0.3.2      R6_2.3.0         rlang_0.3.0.1   
[11] httr_1.3.1       dplyr_0.7.8      tools_3.5.1      yaml_2.2.0       assertthat_0.2.0
[16] tibble_1.4.2     crayon_1.3.4     bindrcpp_0.2.2   tidyr_0.8.2      purrr_0.2.5     
[21] readr_1.1.1      curl_3.2         glue_1.3.0       compiler_3.5.1   pillar_1.3.0    
[26] pkgconfig_2.0.2 

Thanks for the great package!

get_bis doesn't work

Hey!

I am trying to run the code as was presented in the example usage. Somehow I have problems with the get_bis function and following warning message occurs:

library(BIS)
datasets <- get_datasets()
head(datasets, 20)
rates <- get_bis(datasets$url[datasets$name == "Policy rates (monthly)"], quiet = TRUE)
Rows: 39 Columns: 933
── Column specification ───────────────────────────────────────────────────────────────
Delimiter: ","
chr (933): X1, X2, X3, X4, X5, X6, X7, X8, X9, X10, X11, X12, X13, X14, X15, X16, X...

ℹ Use spec() to retrieve the full column specification for this data.
ℹ Specify the column types or set show_col_types = FALSE to quiet this message.
Rows: 38 Columns: 933
── Column specification ───────────────────────────────────────────────────────────────
Delimiter: ","
chr (10): FREQ, Frequency, REF_AREA, Reference area, COMPILATION, Decimals, SOURCE...
dbl (921): DECIMALS, 1946-01, 1946-02, 1946-03, 1946-04, 1946-05, 1946-06, 1946-07,...
lgl (2): TIME_FORMAT, Time Format

ℹ Use spec() to retrieve the full column specification for this data.
ℹ Specify the column types or set show_col_types = FALSE to quiet this message.
Warning messages:
1: as_data_frame() was deprecated in tibble 2.0.0.
ℹ Please use as_tibble() instead.
ℹ The signature and semantics have changed, see ?as_tibble.
ℹ The deprecated feature was likely used in the BIS package.
Please report the issue at https://github.com/expersso/BIS/issues.
This warning is displayed once every 8 hours.
Call lifecycle::last_lifecycle_warnings() to see where this warning was generated.
2: The x argument of as_tibble.matrix() must have unique column names if
.name_repair is omitted as of tibble 2.0.0.
ℹ Using compatibility .name_repair.
ℹ The deprecated feature was likely used in the tibble package.
Please report the issue at https://github.com/tidyverse/tibble/issues.
This warning is displayed once every 8 hours.
Call lifecycle::last_lifecycle_warnings() to see where this warning was generated.

Any help is very much appreciated!

Best reagrds,
Veit