imanuelcostigan / dataonderivatives Goto Github PK

View Code? Open in Web Editor NEW

37.0 37.0 15.0 3.52 MB

Easily source publicly available data on derivatives

Home Page: http://imanuelcostigan.github.io/dataonderivatives/

R 100.00%

cran derivative derivatives r

dataonderivatives's People

Contributors

Stargazers

Watchers

Forkers

ericforgy miceli gitter-badger fkotey ianmadlenya jimhester freephys thalesshannonwatson alexeijiltsov felixmerz vcgato29 eduardobsg davidemagno pythonpowertraders

dataonderivatives's Issues

Get BGCSEF reports

http://dailyactprod.bgcsef.com/SEF/DailyAct/

Suppress messages by default

Rule 11:

Rule of Silence: When a program has nothing surprising to say, it should say nothing.

Get CFTC weekly swaps report

http://www.cftc.gov/MarketReports/SwapsReports/index.htm

Clarify nature of Trade Repository in docs

Principle 24
National regulations: CFTC, EC, etc.
Trade repositories: DTCC, BSDR, CME, Regis-tr. Different subsidiaries of these companies act as repos under different national regs.

Be able to source Tullets data

Here

Improve test coverage

Question: Data Dupes between BSDR and DTCC

This isn't really an issue, more of a question. Do you need to dedupe transactions that may be reported to multiple SDRs? Have you given any thought about how to dedupe if so?

Nice work on putting this together. I've been scraping DTCC data and storing them in my own personal S3 on AWS. If you need an extract or something let me know. Thanks

Format BSEF data results

> str(get_bsef_eod_data('ir', today() - days(1)))
'data.frame':   66 obs. of  12 variables:
 $ tradeDate          : chr  "2014-09-18T00:00:00.000+00:00" "2014-09-18T00:00:00.000+00:00" "2014-09-18T00:00:00.000+00:00" "2014-09-18T00:00:00.000+00:00" ...
 $ security           : chr  "CHF SWAP VS 6M 10YR" "EUR SWAP VS 3M 30YR" "EUR SWAP VS 3M 15YR" "EUR SWAP VS 6M 9YR" ...
 $ priceOpen          : chr  "0.8774999999999999" "1.9925" "1.628" "1.139" ...
 $ priceHigh          : chr  "0.8774999999999999" "1.9925" "1.628" "1.139" ...
 $ priceLow           : chr  "0.8774999999999999" "1.9925" "1.628" "1.139" ...
 $ priceClose         : chr  "0.8774999999999999" "1.9925" "1.628" "1.139" ...
 $ settlementPrice    : chr  "0.8774999999999999" "1.9925" "1.628" "1.139" ...
 $ totalVolume        : chr  "20000000" "180000" "300000" "500000" ...
 $ blockTradeVolume   : chr  "0" "0" "0" "0" ...
 $ totalVolumeUsd     : chr  "21326500" "232056" "386730" "644600" ...
 $ blockTradeVolumeUsd: chr  "0" "0" "0" "0" ...
 $ currency           : chr  "CHF" "EUR" "EUR" "EUR" ...

Create extension of tibble to store data sets

Will allow things like method dispatch and enforce constructor to check that data sets have the right sort of field names and types.

Getting error when requesting sdr data

Hi,

I am new to R. This is my first use of a package so it is possible I am missing something very basic.
Note that I am trying to get transaction level data.

1)When I try to get Bloomberg SDR data, I get the following error:

df1 <- get_bsdr_data(lubridate::ymd(20170109), "CR", 0)
Error: lexical error: invalid char in json text.
request validation fail
(right here) ------^

2)When I try to get DTCC SDR data,, I get the following error:

df1 <- get_ddr_data(lubridate::ymd(20170109), "CR", 0)
Error in return(readr::read_csv(ddrfile[1]), col_types = NULL) :
multi-argument returns are not permitted
In addition: Warning message:
url_ok(x) is deprecated; please use identical(status_code(x), 200L) instead.

Can you please help me understand the issue here? Your help will be greatly appreciated!

Thanks!

bsdr(lubridate::ymd(20150504), "IR")
#> Error in curl::curl_fetch_memory(url, handle = handle) : 
#>  Proxy CONNECT aborted
#> Run `rlang::last_error()` to see where the error occurred.

Return NULL when data doesn't exist

This will allow getter to be called in loop with result DFs being bound like this:

df <- NULL
for (date in dates) {
  df <- bind_rows(df, get_ddr_data(date))
}

bind_row(NULL, df) returns NULL where df is NULL and df otherwise.

# ICAP US 31/7/2014
head -n 1 /var/folders/9f/kz67r0rn3qb9tx_rmgbps94r0000gq/T//Rtmp7b11UG/icus/*20140731*
BATCHDATE,TRADEINSTRID,ASSETCLASS,Trade Volume (USD),Trade Volume (Local Currency),OPENPRICE,HIGHPRICE,LOWPRICE,CLOSEPRICE
# IGDL 31/7/2014
head -n 1 /var/folders/9f/kz67r0rn3qb9tx_rmgbps94r0000gq/T//Rtmp7b11UG/igdl/*20140731*
BACTHDATE,TRADEINSTRID,ASSET CLASS,TRADEVOLUME USD,TRADEVOLUME,OPENPRICE,HIGHPRICE,LOWPRICE,CLOSEPRICE

Example of different headers for the same file but on different dates:

head -n 1 /var/folders/9f/kz67r0rn3qb9tx_rmgbps94r0000gq/T//Rtmp7b11UG/igdl/*20140630*
BATCHDATE,TRADEINSTRID,ASSETCLASS,Trade Volume (Local Currency),Trade Volume (Local Currency),OPENPRICE,HIGHPRICE,LOWPRICE,CLOSEPRICE
head -n 1 /var/folders/9f/kz67r0rn3qb9tx_rmgbps94r0000gq/T//Rtmp7b11UG/igdl/*20140701*
BATCHDATE,TRADEINSTRID,ASSETCLASS,Trade Volume (USD),Trade Volume (Local Currency),OPENPRICE,HIGHPRICE,LOWPRICE,CLOSEPRICE

DTCC URL scheme changes sometime in late November

We see the following:

ddr(lubridate::ymd(20201130), "IR")
#> # A tibble: 0 x 0

Be able to source Bloomberg SEF data

BSEF data

Use standard evaluation semantics rather than non-standard evaluation

I.e. don't do this:

df %>%
      dplyr::mutate(date = lubridate::ymd_hms(tradeDate),
        assetclass = factor(assetclass))

Use mutate_ instead. Means I don't need to declare global variables.

Be able to get SDR data

Bloomberg SDR
DTCC
CME has anon FTP access